search

sukrutrao / Fast-Dawid-Skene

Code for the algorithms in the paper: Vaibhav B Sinha, Sukrut Rao, Vineeth N Balasubramanian. Fast Dawid-Skene: A Fast Vote Aggregation Scheme for Sentiment Classification. KDD WISDOM 2018
https://sites.google.com/view/fast-dawid-skene
MIT License
42 stars 11 forks source link

Variable number of labelers? #1

Closed fedor57 closed 6 years ago

fedor57 commented 6 years ago

Hi, I tried to test the algorithm over some real-life data. I found the limitation of having a fixed number of labelers a little bit impractical. I wonder if this is a particular implementation limitation or one of the aggregation model? What if a user provide some kind of reasonable upper limit for a task labelers count (3-5-7)? Thank you!

sukrutrao commented 6 years ago

Hi, Thanks for your question. It is a limitation of the implementation, not of the model. We will update it to allow for a variable number of annotators per question.

sukrutrao commented 6 years ago

Hi, This has been updated in #2. Now, setting k=0 or not setting k at all will make the algorithm use all available annotations for each data point.

Please re-open the issue if any problems persist. Thank you!

fedor57 commented 6 years ago

Fantastic, thank you!!! Gonna test it and let you know!