search

proycon / flat

FoLiA Linguistic Annotation Tool -- Flat is a web-based linguistic annotation environment based around the FoLiA format (http://proycon.github.io/folia), a rich XML-based format for linguistic annotation. Flat allows users to view annotated FoLiA documents and enrich these documents with new annotations, a wide variety of linguistic annotation types is supported through the FoLiA paradigm.
GNU General Public License v3.0
111 stars 15 forks source link

Adjudication for FLAT-based annotations #148

Open luutuntin opened 5 years ago

luutuntin commented 5 years ago

Hi Maarten,

This relates to #112. In addition to IAA calculation, we need an adjudication interface to compare different annotations and create a gold standard. What are the adjudication practices of FLAT-based annotation projects that you are aware of?

Best, Alex

OsmanMutlu commented 5 years ago

Hi Alex,

You can use the merging script provided by proycon to merge annotations from different annotators into one file, then make changes on it in FLAT.

proycon commented 5 years ago

We don't really have an interface for adjudication yet in FLAT, this is mostly deferred to application-specific postprocessing scripts, such as the foliaagreement tool (just a computation without interface though) or through foliamerge as @OsmanMutlu suggested. Sometimes you can also leverage the power of FLAT to do explicit corrections on annotations, if appropriate for your project.

luutuntin commented 5 years ago

Thank you, Osman and Maarten. I got the idea now.

I tried foliamerge (folia-2.1.4 and folia-tools-2.1.6) and received the following error:

File "d:\users\luutuntin\anaconda3\envs\py36\lib\site-packages\foliatools\foliamerge.py", line 41, in mergechildren if isinstance(e, (folia.AbstractTokenAnnotation, folia.AbstractAnnotationLayer)) and parent.id and not hasattr(e, 'merged'): AttributeError: module 'folia.main' has no attribute 'AbstractTokenAnnotation'

I couldn't find foliaagreement in FoLiA tools, I tried foliaeval (but I was not sure what is the correct way to provide the value for TYPE argument, e.g. POS or pos-annotation?) and received the following error:

File "d:\users\luutuntin\anaconda3\envs\py36\lib\site-packages\foliatools\foliaeval.py", line 380, in main evaluation = evaluate(docs, Type, foliaset, False, args.corrections, args.confusionmatrix, args.verbose) UnboundLocalError: local variable 'Type' referenced before assignment

Did I ignore certain important details?

proycon commented 5 years ago

I tried foliamerge (folia-2.1.4 and folia-tools-2.1.6) and received the following error.

Hmm, indeed something broke when the new library came out. I just committed a fix and will do a new release.

I couldn't find foliaagreement in FoLiA tools, I tried foliaeval (but I was not sure what is the correct way to provide the value for TYPE argument, e.g. POS or pos-annotation?) and received the following error:

Oops, I indeed meant foliaeval. The type value is equal to the FoLiA XML tag, so in this case pos.