search

ipeirotis / Get-Another-Label

Quality control code for estimating the quality of the workers in crowdsourcing environments
70 stars 26 forks source link

Add weighted summaries in the summary, for the worker quality metrics #15

Closed ipeirotis closed 12 years ago

ipeirotis commented 12 years ago

Right now we display the averages of worker quality, assuming that each worker is equal to others. However, since each worker is contributing different number of labels, we need to also report the weighted average of worker quality. For that, we multiply each quality metric with the "number of annotations" (aka assigned labels), and then report the sum of these products, divided by the total number of labels assigned by workers

aldrinleal commented 12 years ago

See comments on #18.

aldrinleal commented 12 years ago

Panagiotis,

This one needs some work from you. Could you reach me to help me assess the ground clearly?

Thanks

ipeirotis commented 12 years ago

Example:

We have: Worker 1, Quality=60%, AssignedLabels=500 Worker 2, Quality=100%, AssignedLabels=100

Non-weighted Quality = (60+100)/2 = 80% Weighted Quality = (60 * 500 + 100 * 100) / (500 +100) = 66.66%

aldrinleal commented 12 years ago

[Weighted Quality (Est Quality Exp)] Weighted Quality w/ Expected cost, according to the algorithm estimates: 31.11% [Weighted Quality (Est Quality Opt)] Weighted Quality w/ Minimized cost, according to the algorithm estimates: 26.67% [Weighted Quality (Eval Quality Exp)] Weighted Quality w/ Expected cost, according to the evaluation data: N/A [Weighted Quality (Eval Quality Opt)] Weighted Quality w/ Minimized cost, according to the evaluation data: N/A