Open DheerajMaddaR opened 4 months ago
Apologies for my late reply!!! I should have enabled notifications for my GitHub repos!
Regarding your question, we follow the conventional practice for evaluating OOD detection, using both FPR95 and AUROC (AUC in our paper).
AUROC evaluates the overall ability of the method to distinguish between positive and negative classes. It doesn't require a specific threshold. Intuitively, AUROC tests the model's performance across all possible thresholds and provides an average measure of its effectiveness. You can find more details here: ROC Curve on Wikipedia.
FPR95 measures the false positive rate when the true positive rate (TPR) for ID samples is 95%. Essentially, it needs a threshold where TPR for ID samples is around 95%.
In our code, please refer to lines 23-35 in utils/metrics.py: https://github.com/Qinyu-Allen-Zhao/OptFSOOD/blob/c09a85e1f15de5a1f45c89419be6a5fff03f88c5/utils/metrics.py#L23-L35
We set recall_level_default=0.95
, use sklearn.metrics.roc_curve
to compute FPR and TPR at different thresholds, and then obtain the FPR at the smallest TPR >= 0.95. If you'd like, you can print out the thresholds to see the details.
When writing this code, I referenced some of the implementations from OpenOOD, which you might also find useful. Their code is a bit more complex than ours, but you can check it out here https://github.com/Jingkang50/OpenOOD/blob/3d7c483d88f15b514ea051799dde4a0cdce30d9c/openood/evaluators/metrics.py#L29-L36
I hope this helps! 😊
Best, Qinyu
At inference, we need a scalar value as threshold to identify the input image is ID or OOD. I went through the code and I did not find any function to compute this threshold value or may be I missed something. Can you please tell how to detect a input sample at inference as OOD or not.
Thankyou for your amazing work!