sanagh
commented
1 year ago Hi
As we have 21 different entity types, we have calculated Cohen's kappa for each entity type separately. Thus, through the calculation of Cohen's kappa with "O" for any entity type like "ORG", we considered all other entity types in addition to the "O" label to be in disagreement with "ORG".
When we calculated Cohen's kappa without "O", we excluded only the agreement of the "O" label, as we mentioned in section 3.4 in the paper. So the value of TN will be the agreement of non-being "ORG", i.e. (to be from the other 20 entity types).
mohammedkhalilia
Thanks for your paper. I have a question about the calculation of kappa without including "O" label in agreement, i.e. $\kappa_{\sim o}$. Let me show an example to let you know my concern.
In Table 6 in the paper, the "ORG" entity tag contains 1713 TP, 33 FN and 30 FP, and as you mentioned, the total num of annotated tokens is 24K (first paragraph in Sec 3.4), so TN (true negative) is $24000-1713-33-20=22224$
If you include "O" in agreement between two annotations, then $TN=22224, TP=1713, FN=33, FP=30, totalnum=24000$. So, $p_o$ in $\kappa_o$ is $(1713+22224)/24000=0.997375$, and $p_e=((1713+30)\times(1713+33)+(22224+30)\times(22224+33))/(24000^2)\approx 0.86519$, so $\kappa_o=(p_o-p_e)/(1-p_e)\approx 0.980528 \approx 0.981$, which is consistent with your result in Table 6.
However, when we exclude "O", then TN here should be 0, and simply use $TN=0, TP=1713, FN=33, FP=30, totalnum=1776$ to compute $\kappa_{\sim o}$ will give us a negative result ($-0.018015$), since change TN to 0 will greatly increase the baseline in Kappa, i.e. $p_e$. This problem is common for computing cohen's kappa in NER setting since the "negative samples" is hard to define: using all "O" may return biased result since the data is imbalanced, while excluding "O“ will give negative result unexpextedly.
So, I want to ask the question: How do you compute $\kappa_{\sim o}$ here, as in Table 6 you claimed that of "ORG" tag is 0.974 and so on ? How do you define $p_o$ and $p_e$ here when excluding "O" ? Can you give me a concrete definition equation?