Determine guidance for a credibility threshold on first party reports

responsible-ai-collaborative / aiid

The AI Incident Database seeks to identify, define, and catalog artificial intelligence incidents.

https://incidentdatabase.ai

Other

167 stars 35 forks source link

Determine guidance for a credibility threshold on first party reports #2063

Open Janetbananet opened 1 year ago

Janetbananet commented 1 year ago

When an editor reviews a first party report submission, he/she will need to evaluate the credibility of the claimed harm in order to decide if the submission should be ingested. We need to define some criteria for evaluating this credibility and publish it so that it is transparent to submitters why their first party reports have/have not been added to the AIID.

Criteria may include:

Completeness of data
Reporting from other sources
TBD

smcgregor commented 1 year ago

Quick additions

Identifiability of the submitter (not a hard requirement, but one that would help)
Technical plausibility
Thematic consistency with better substantiated incidents
Multiplicity of reports with consistent elements
...

Janet-ResponsibleAI commented 6 months ago

Whew, this one is going to be a doozy, but it's important. I'll keep tabs on it as we continue discussions on editorial items.

lmcnulty commented 6 months ago

This might be related to https://github.com/responsible-ai-collaborative/aiid/issues/1638.