search

cproctor / qualitative-coding

Qualitative coding for computer scientists
Other
9 stars 3 forks source link

Implement new CLI command for inter-rater agreement #21

Open cproctor opened 8 months ago

cproctor commented 8 months ago

And look into some metrics we may want to implement. Consider Shaffer's RhoR and some of his critiques of inter-rater agreement practices.

cproctor commented 8 months ago

After reading [1], I propose that we implement Kappa, J, Precision, Recall, and F statistics. They're quite straightforward to implement. I also think we should implement Shaffer's rho, which is described in Quantitative Ethnography (I have a copy in my office).

Let's start with kappa, and leave the rest as enhancements.

[1] Eagan, B., Brohinsky, J., Wang, J., & Shaffer, D. W. (2020). Testing the reliability of inter-rater reliability. Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, 454–461. https://doi.org/10.1145/3375462.3375508

cproctor commented 2 days ago

I think we should use this paper as the methodological basis for implementing agreement:

Halpin, S. N. (2024). Inter-Coder Agreement in Qualitative Coding: Considerations for its Use. American Journal of Qualitative Research, 8(3), 23–43. https://doi.org/10.29333/ajqr/14887