search

tnc-ca-geo / animl-frontend

A frontend web app for viewing & labeling camera trap data by The Nature Conservancy.
https://animl.camera
Other
16 stars 0 forks source link

Audit feature for QA/QC of labels #86

Open nathanielrindlaub opened 2 years ago

nathanielrindlaub commented 2 years ago

Develop this w/ Juli and Nick. Nick has suggested a couple ideas: one is a random selection of validated rodents and double check those, or “plant” a handful of rat images in the batch before a big label sprint, and see what % get flagged.

Most thorough would be to have some kind of “super confirm” / “retire” level of validation, and only high-permission level users would be able to do it, and Juli or some expert would be responsible for retiring ALL images labeled by volunteers.

If Juli doesn’t have capacity, maybe something along the lines of click a button and get a random sample?

nathanielrindlaub commented 1 year ago

Splitting the concept of a randomized "Audit" feature out from a secondary review feature, which is now covered in #137.

The Audit feature would be some kind of button or view in which project managers or experts are presented with some number random reviewed images, which they then must "Retire" if labeled correctly or fix if not, and at the end of the review they're given stats about the % of correctly labeled images for each label & perhaps some indication of which users were mislabeling them