Krippendorff's alpha_U for empty annotation sets or a single annotation unit

[chmeyer]

Originally reported on Google Code with ID 2

- What steps will reproduce the problem?

@Test
public void testOnlyOneAnnotates()
{
    UnitizingStudy study = new UnitizingStudy(3);
    study.addSection("X", 0, 1, 1);
    study.setContinuumLength(20);
    study.close();

    AlphaUnitizedAgreement agreement = new AlphaUnitizedAgreement(study);

    assertTrue(agreement.estimateJointAgreement() < 1.0);
}

- What is the expected output? What do you see instead?

Gives 1.0 agreement; but from three annotators, only one annotates something; the other
two did not annotate anything. I would expect some much lower number...

- What version of the product are you using? On what operating system?

1.0.2

- Please provide any additional information below.

Reported by ivan.habernal on 2014-04-04 13:56:59

[chmeyer]

Good morning,
The reason for the perfect agreement is that the observed disagreement is 0.
This results from annotator 1 and 2 not having any sections (not even gap sections).

I guess this is a bug, because unannotated sections should be taken into consideration
while calculating the observed disagreement.

The fix should be done in UnitizingStudy.close, I guess. I will have a look at that
later.

Regards,
Roland

Reported by roland.kluge.de on 2014-04-05 06:15:08

• Status changed: Accepted

[chmeyer]

The test case runs now. The reason was indeed that annotators without any annotations
were ignored in the UnitizedStudy#close() method.

Regards,
Roland

Reported by roland.kluge.de on 2014-04-06 11:30:00

• Status changed: Fixed

[chmeyer]

Hi Roland, thanks for fixing this! I have another (a kind of) border case: what if all
three annotators agree there is no annotation in the document? Naively, I would assume
it as a perfect agreement. Does unitized alpha cover that as well?

Some test code:

@Test
public void testNobodyAnnotates()
{
    UnitizingStudy study = new UnitizingStudy(3);
    study.setContinuumLength(20);
    study.close();

    AlphaUnitizedAgreement agreement = new AlphaUnitizedAgreement(study);
    double result = agreement.estimateJointAgreement();

    assertEquals(1.0, result, 0.001);
}

Best,

Ivan

Reported by ivan.habernal on 2014-04-07 07:10:10

[chmeyer]

I have added the two test cases to de.tudarmstadt.ukp.dkpro.statistics.agreement.unitizing.UnitizingAgreementTest
(thanks for providing them!).

Regarding the case of having no annotations at all: it is not possible to calculate
inter-rater agreement for this case. So it is, by definition, not perfect agreement,
but an undefined (in the software library, this is represented by NaN). The rationale
behind this is that there is neither observed disagreement nor expected disagreement.

Assuming perfect agreement is an option, but I would be careful with this conclusion,
since it could easily be the case that the experimental setup is flawed. Assume you
want to mark argument structures in a text and multiple annotators agree that there
are none. This essentially tells you that the text you have chosen does not contain
argument structures or that your guidelines are unclear, but it does not tell you anything
about the difficultly of annotation argument structures, since you cannot measure any
variance for the NON-GAP category of your annotation study.

(BTW: sorry for late response, e-mail notification was turned off :-/).

Reported by chmeyer.de on 2014-07-25 14:16:56

• Labels added: Component-Agreement

[chmeyer]

Reported by chmeyer.de on 2014-08-20 16:16:59

• Labels added: Milestone-2.0.0