Though it's probably not one of the better measures of interannotator agreement, I have frequently seen pairs of set-valued annotations compared by means of the Dice coefficient: 2 × | A ∩ B | ÷ (|A| + |B|) The coefficients are then averaged across all instances.
It would be nice if DKPro Statistics were able to reproduce such measurements.
Though it's probably not one of the better measures of interannotator agreement, I have frequently seen pairs of set-valued annotations compared by means of the Dice coefficient: 2 × | A ∩ B | ÷ (|A| + |B|) The coefficients are then averaged across all instances.
It would be nice if DKPro Statistics were able to reproduce such measurements.