Open nathanielrindlaub opened 2 years ago
Splitting the concept of a randomized "Audit" feature out from a secondary review feature, which is now covered in #137.
The Audit feature would be some kind of button or view in which project managers or experts are presented with some number random reviewed images, which they then must "Retire" if labeled correctly or fix if not, and at the end of the review they're given stats about the % of correctly labeled images for each label & perhaps some indication of which users were mislabeling them
Develop this w/ Juli and Nick. Nick has suggested a couple ideas: one is a random selection of validated rodents and double check those, or “plant” a handful of rat images in the batch before a big label sprint, and see what % get flagged.
Most thorough would be to have some kind of “super confirm” / “retire” level of validation, and only high-permission level users would be able to do it, and Juli or some expert would be responsible for retiring ALL images labeled by volunteers.
If Juli doesn’t have capacity, maybe something along the lines of click a button and get a random sample?