search

jmgirard / mReliability

MATLAB Functions for Computing Inter-Observer Reliability
http://mreliability.jmgirard.com
GNU General Public License v2.0
40 stars 8 forks source link

It seems the code for kappa is wrong #2

Closed AlessandroChecco closed 7 years ago

AlessandroChecco commented 7 years ago

I tested mKAPPA with one document and 5 raters and the results are very strange 

mKAPPA([1,1,1,1,1],[1 2],'ordinal')
ans =
   NaN

(while I would expect to get 1)

and 

mKAPPA([1,1,1,1,2],[1 2],'ordinal')
ans = 
  -2.7756e-16

(while I would expect to get a number close to 1)

AlessandroChecco commented 7 years ago

I close this because these degenerate cases are not handled well by Kappa index, not by your code.

jmgirard commented 7 years ago

Hi Alessandro, You're correct. This is an issue with the kappa formulation as opposed to the mKAPPA function. In your case, I might recommend using mSCOTTPI. Please let me know if you have any other issues or questions.

AlessandroChecco commented 7 years ago

Thank you very much for your answer. I'll look into it!

AlessandroChecco commented 7 years ago

mSCOTTPI([1,1,1,1,1],[1,2],'ordinal') ans = NaN

mSCOTTPI([1,1,1,1,2],[1,2],'ordinal') ans = -0.2500

mSCOTTPI([1,1,1,2,2],[1,2],'ordinal') ans = -0.2500

mSCOTTPI([1,1,2,2,2],[1,2],'ordinal') ans = -0.2500

mSCOTTPI([1,2,2,2,2],[1,2],'ordinal') ans = -0.2500

mSCOTTPI([1,2,2,2,2],[1,2],'ordinal') ans = -0.2500

It seems that for one document and 5 raters this kind of systems doesn't make sense...