Annotation Design Doc

[ejhumphrey]

Addresses the following (non-exhaustive list):

• what motivates annotators
  • cost of open-mic participation?
  • monetary reward?
  • citizen science?
• who can annotate (open-mic participants? anyone?),
• how to control for quality (depends on who can annotate)
• what kind of feedback do annotators get?
  • Does annotation work get logged somewhere (e.g. in case we want to add attribution or gamification further down the line)?
  • Can the annotator access these stats?
• Does the annotator have any control over the type of audio content they are served?

[ejhumphrey]

A (mostly unpopulated) document for this now lives here: https://docs.google.com/document/d/1_1odgrnakmVKQcK0gEn72kqX3Xq-1eyh0j802hhEnI4/edit?usp=sharing

[justinsalamon]

ok cool, I see you've moved my comments from the annotation system gdoc over here and set up a new gdoc for this.

@ejhumphrey do you want to throw some ideas around here first, or shall we work directly on the gdoc?

[ejhumphrey]

let's iterate in the doc. We're starting from basically zero and I anticipate enough iteration that it'd render this thread unintelligible quickly.

[justinsalamon]

👍

[ejhumphrey]

Will echo these thoughts in the doc, but it came up in conversation today that we might be best served by using a variety of annotation strategies, both concurrently and over time, i.e. cold annotation to start, identify instrument experts and present as a retrieval task (which of these have X), etc

[justinsalamon]

@ejhumphrey I've added some comments to the annotation process design doc, but also a bunch of considerations under "How can we control for annotation quality? / DISCUSSION". Don't think google pings about text additions, so just wanted to draw your attention to this.