search

savkov / bratutils

A collection of utilities for manipulating data and calculating inter-annotator agreement in brat annotation files.
MIT License
29 stars 12 forks source link

Apply to attributes and relations too #14

Closed soluna1 closed 5 years ago

soluna1 commented 5 years ago

Hi Savkov,

Knowing this tool before would have saved me lot of time. I used NLTK package to measure IAA of brat annotation files. A bit a nightmare to convert the "ann" file to something readable. So, I think that this tool is very useful, and the code is great, congratulations!

Our problem is that we have data structured in this way:

T1 Food 24 31 bacalao T2 Restaurant 0 8 Un sitio T3 Restaurant 46 54 Un lugar T5 Restaurant 55 66 con encanto A3 Polarity T5 POS A4 Restaurant_Aspects T5 General_experience R2 refers_to Arg1:T5 Arg2:T3
T4 Food 34 43 riquísimo A1 Polarity T4 POS A2 Food_Aspects T4 General_experience R1 refers_to Arg1:T4 Arg2:T1

And we want to measure agreement for the 3 categories, entities (Food#Bacalao), attribute ( aspect -->General_experience#con encanto; and polarity --> POS), and also relationships (R1 refers_to...). Are you planning to implement these options too? It would be really useful for annotation at aspect-based Sentiment Analysis.

Many thanks

jeanphilippegoldman commented 5 years ago

I had this in mind at the exact same minute : attributes, relations...AND...discontinued annotations. At least it should ignore them so it does not fail.

(Soluna, could you point me to the converting scripts so .ann files can be read with NLTK please ?)

Many thanks

savkov commented 5 years ago

Hey, this has been requested multiple times. It's not trivial to implement. I've made some assumptions in the code that make things easier for me because I didn't need to support those at the time. It would be great if someone implements it but it's very unlikely that that'd be me -- too many weekend projects.

savkov commented 5 years ago

Closing as there will be better-suited issues for this. Like #17 and #16