search

DeveloperLiberationFront / AffectAnalysisToolEvaluation

SEmotion_18 paper on evaluating the reliability of sentiment and politeness analysis tools
3 stars 4 forks source link

Human bias in manual coding #12

Closed nasifimtiazohi closed 6 years ago

nasifimtiazohi commented 6 years ago

R1

4 - Another concern is about the rating process, among the four rater, Coder 1 rated all comments and later discuss with other raters to resolve conflicts on rated comments. This is a "human bias" because Coder 1 rated all comments in the dataset and may had influenced other raters.

nasifimtiazohi commented 6 years ago

this is definitely a major threat.

mentioned in threats.

https://github.com/DeveloperLiberationFront/AffectAnalysisToolEvaluation/commit/5fe013734f1a390f393ce819a6b6c1c73de673c4

nasifimtiazohi commented 6 years ago

also, we have results over the initially agreed comments where both the coder gave the same label without having any prior discussion.

added as an explanation,

https://github.com/DeveloperLiberationFront/AffectAnalysisToolEvaluation/commit/1b919d84c67c372ddd12c23d96fb4163e34369b5