analyze pre-study results (and flag bad turkers)

[bellecarrell]

flagging bad turkers seems to be the primary goal of analyzing results

From Adrian: The first batch of annotations is collected. Could you compute some basic statistics over these results:

• [x] % account type label (treating each annotation separately, majority vote, and unanimous vote)
• [x] distribution over category labels ( "" )

Also for quality control can you compute the following statistics over all workers:

• [x] Mean & median time to complete assignment
• [x] Number of assignments completed
• [x] Inter-annotator agreement for each label (averaged over pairs of other workers)
• [x] Proportion users labeled as non-promotional

We will use these values to decide which workers to drop. We have 3 days to reject Turkers who were not taking this seriously. Let me know if you have any questions on how to do any of the above.

• [x] calculate statistics
• [ ] flag any bad turkers

[bellecarrell]

going to return after lunch + gym to finish

[bellecarrell]

@abenton intermediate results in /intermediate/ dir under /uis/hits/ will finish tomorrow asap missing: inter-annotator, proportion users labeled non-promotional, distribution for all category labels

[bellecarrell]

@abenton stats pushed. let me know if you have any questions. all of the category_all percentages were calculated with a total value equaling the number of assignments labeled as promoting

[bellecarrell]

@abenton updated csvs are in the repo

[bellecarrell]

no bad turkers ided. more serious iaa stats coming