anssiko
commented
2 years ago Thank you @tobie for your suggestions. I suspect you found this document through its Draft Note publication announcement which provides some validation that pushing the first draft out to TR helped reach more folks outside the WG. That is, TR works as designed.
On the topic, we'll discuss this issue when we do out next issue triage, most likely after the vacation period. There's also a plan to have a session around ethics & ML at TPAC and you're welcome to join that discussion if you're interested.
tobie
Section 3.1 on accuracy states, referencing Dr. Leslie's paper, that "[t]he accuracy of an ML model is the proportion of examples for which it generates a correct output".
Dr. Leslie goes on to characterize the different kinds of incorrect output and their varying context-dependent impact.
This feels important to outline in this document.
For example, when identifying cancerous skin growth, false positives may slightly increase the cost of cancer detection (by requiring additional lab work to rule them out), while false negatives may delay treatment to the point where it is no longer effective. What matters here is lowering the risk of false negatives as much as possible.
In a judicial context, false positives may send innocents behind bars for decades, while false negatives might sometimes make it more difficult to convict a criminal. What matters here is lowering the risk of false positives entirely.