search

dkpro / dkpro-statistics

DKPro Statistics
https://dkpro.github.io/dkpro-statistics
Apache License 2.0
12 stars 4 forks source link

NaN result of expected agreement matches actual agreement #13

Closed chmeyer closed 9 years ago

chmeyer commented 9 years ago

Originally reported on Google Code with ID 13

If the expected agreement (EA) matches the actual agreement (AA), the final result is
NaN. E.g. if EA=1 and AA=1, the result is NaN. Shouldn't the result be 1 in this case
- or in general if EA = AA, shouldn't the result be the value of AA?

Observed in DKPro Statistics 2.0.0 using CohenKappaAgreement on a study containing
only one item on which the annotators agreed.

Reported by richard.eckart on 2015-06-18 09:11:13

chmeyer commented 9 years ago 
This is a difficult case, see issue 1.

Returning 1.0 as chance-corrected agreement is not correct, IMHO. I leave the issue
open, such that documentation can be improved.

Reported by chmeyer.de on 2015-06-18 09:17:37

chmeyer commented 9 years ago 
Ok. I had this in a unit test and worked around it by simply an adding additional non-different
line to the study. 

Still, maybe throwing some "InsufficientDataException" would be sensible.

Reported by richard.eckart on 2015-06-18 09:36:08

chmeyer commented 9 years ago

Fixed in version 2.1.0. The solution raises the proposed InsufficientDataException.