Inter-annotator agreements

[twinkarma]

• Support calculator IAA for

  • Feature where you can apply a nominal, ordinal or continuous scale i.e. only single selectable option
  • Radio buttons, Selector, Checkboxes where you can only select a single option
  • Multiple choice to be supported later, we'd require set arithmatic

• Extract for each pair of annotators all the annotations they have done

• Calculate simple matches if doc classification

• For sequence labelling, we need to do this with begin/end offsets and per annotation type

[davidwilby]

discuss with Xingyi (@deansong), he has scripts for doing this

[davidwilby]

rules in old teamware are same as rules in GATE developer

[twinkarma]

• Check with team for existing inter-annotator agreement code.
• Also ask the team for testing data to double check this

[twinkarma]

Common remedial action

• Remove all annotator's annotation from project, remove from project
• Revise guideline for all annotators (will start new project in this case)

[ianroberts]

Things to consider - different types of IAA metrics for different types of task

• binary / "nominal" classification - exact match is "correct", anything else is "wrong"
• Likert-type "ordinal" scales with a finite number of options which are ordered, so e.g. 4 vs 5 is "closer" than 4 vs 1
• continuous numerical scales

Maybe this needs to be part of the project configuration for each widget, so we know what sort of metrics to use for that particular question.

[ianroberts]

Variables may not be independent, e.g. a classification plus associated confidence score - do we want to use the confidence score as a weighting of some kind or as a threshold (e.g. only compute IAA on pairs where both annotators confidence is > 4)

[twinkarma]

We need to check what the standard procedure is for comparing multiple choice agreement

[davidwilby]

Depends upon defined scale for a particular project #349