Closed chmeyer closed 9 years ago
Good morning,
The reason for the perfect agreement is that the observed disagreement is 0.
This results from annotator 1 and 2 not having any sections (not even gap sections).
I guess this is a bug, because unannotated sections should be taken into consideration
while calculating the observed disagreement.
The fix should be done in UnitizingStudy.close, I guess. I will have a look at that
later.
Regards,
Roland
Reported by roland.kluge.de
on 2014-04-05 06:15:08
Accepted
The test case runs now. The reason was indeed that annotators without any annotations
were ignored in the UnitizedStudy#close() method.
Regards,
Roland
Reported by roland.kluge.de
on 2014-04-06 11:30:00
Fixed
Hi Roland, thanks for fixing this! I have another (a kind of) border case: what if all
three annotators agree there is no annotation in the document? Naively, I would assume
it as a perfect agreement. Does unitized alpha cover that as well?
Some test code:
@Test
public void testNobodyAnnotates()
{
UnitizingStudy study = new UnitizingStudy(3);
study.setContinuumLength(20);
study.close();
AlphaUnitizedAgreement agreement = new AlphaUnitizedAgreement(study);
double result = agreement.estimateJointAgreement();
assertEquals(1.0, result, 0.001);
}
Best,
Ivan
Reported by ivan.habernal
on 2014-04-07 07:10:10
I have added the two test cases to de.tudarmstadt.ukp.dkpro.statistics.agreement.unitizing.UnitizingAgreementTest
(thanks for providing them!).
Regarding the case of having no annotations at all: it is not possible to calculate
inter-rater agreement for this case. So it is, by definition, not perfect agreement,
but an undefined (in the software library, this is represented by NaN). The rationale
behind this is that there is neither observed disagreement nor expected disagreement.
Assuming perfect agreement is an option, but I would be careful with this conclusion,
since it could easily be the case that the experimental setup is flawed. Assume you
want to mark argument structures in a text and multiple annotators agree that there
are none. This essentially tells you that the text you have chosen does not contain
argument structures or that your guidelines are unclear, but it does not tell you anything
about the difficultly of annotation argument structures, since you cannot measure any
variance for the NON-GAP category of your annotation study.
(BTW: sorry for late response, e-mail notification was turned off :-/).
Reported by chmeyer.de
on 2014-07-25 14:16:56
Reported by chmeyer.de
on 2014-08-20 16:16:59
Originally reported on Google Code with ID 2
Reported by
ivan.habernal
on 2014-04-04 13:56:59