Open mattbit opened 1 year ago
This was mostly adressed in #1193, althought the Benjamini–Hochberg procedure is not enabled by default (because statistical tests on metrics like balanced accuracy pose problems).
Not completed yet
Hello,
It's KD_A from Reddit. I purged my account recently, so the linked Reddit comment is no longer available. Posting it and the next reply here for posterity:
Following the feedback by user KD_A on reddit. They recommend more sound handling of statistical significance to prevent selection bias, in particular using a Benjamini-Hochberg procedure to control the false discovery rate.
The problem is that we currently test several data slice candidates + metric without accounting for selection bias → this can lead to a high number of false positive detections.
To do
PerformanceBiasDetector
and filter the detections based on their p-value with Benjamini-Hochberg procedure.From SyncLinear.com | GSK-1279