Gold standard - based worker evaluation

[GoogleCodeExporter]

When a worker requests a particular task, they will first get gold standard
HITs to measure their performance. If their performance is too low, they
won't be allowed to work on the actual session HITs.

Original issue reported on code.google.com by syrn...@gmail.com on 18 Jan 2010 at 2:43

[GoogleCodeExporter]

Original comment by syrn...@gmail.com on 22 Jan 2010 at 4:06

• Added labels: Priority-High
• Removed labels: Priority-Critical

[GoogleCodeExporter]

Original comment by syrn...@gmail.com on 22 Jan 2010 at 7:31

• Added labels: Component-MturkBase, Priority-Medium
• Removed labels: Priority-High

[GoogleCodeExporter]

Partially implemented. The gold session can have multiple submissions and the 
target
session would substitute the gold items according to a schedule. The 
substitution
creates ItemSubstitution records that will record that the actual assignments 
were
spent on doing something different.

The item substitutions need to be cleared at some point and consumed assignments
resubmitted.

Original comment by syrn...@gmail.com on 30 Mar 2010 at 10:59

[GoogleCodeExporter]

Need to set accuracy limits on the session - when will the workers get blocked 
and
how can they request permission to work on the data further.

Original comment by syrn...@gmail.com on 30 Mar 2010 at 11:01

[GoogleCodeExporter]

Created gold standard-based worker evaluation. We keep an average worker score 
and
block the worker if they fall below the threshold.

Original comment by syrn...@gmail.com on 30 Mar 2010 at 11:28

• Changed state: Fixed