Qinyu-Allen-Zhao
commented
1 month ago Apologies for my late reply!!! I should have enabled notifications for my GitHub repos!
Regarding your question, we follow the conventional practice for evaluating OOD detection, using both FPR95 and AUROC (AUC in our paper).
-
AUROC evaluates the overall ability of the method to distinguish between positive and negative classes. It doesn't require a specific threshold. Intuitively, AUROC tests the model's performance across all possible thresholds and provides an average measure of its effectiveness. You can find more details here: ROC Curve on Wikipedia.
-
FPR95 measures the false positive rate when the true positive rate (TPR) for ID samples is 95%. Essentially, it needs a threshold where TPR for ID samples is around 95%.
In our code, please refer to lines 23-35 in utils/metrics.py: https://github.com/Qinyu-Allen-Zhao/OptFSOOD/blob/c09a85e1f15de5a1f45c89419be6a5fff03f88c5/utils/metrics.py#L23-L35
We set recall_level_default=0.95, use sklearn.metrics.roc_curve to compute FPR and TPR at different thresholds, and then obtain the FPR at the smallest TPR >= 0.95. If you'd like, you can print out the thresholds to see the details.
When writing this code, I referenced some of the implementations from OpenOOD, which you might also find useful. Their code is a bit more complex than ours, but you can check it out here https://github.com/Jingkang50/OpenOOD/blob/3d7c483d88f15b514ea051799dde4a0cdce30d9c/openood/evaluators/metrics.py#L29-L36
I hope this helps! 😊
Best, Qinyu
At inference, we need a scalar value as threshold to identify the input image is ID or OOD. I went through the code and I did not find any function to compute this threshold value or may be I missed something. Can you please tell how to detect a input sample at inference as OOD or not.
Thankyou for your amazing work!