Qinyu-Allen-Zhao / OptFSOOD

Code for ICLR 2024 paper "Towards Optimal Feature-Shaping Methods for Out-of-Distribution Detection"
12 stars 2 forks source link

How to detect as OOD sample during inference #1

Open DheerajMaddaR opened 4 months ago

DheerajMaddaR commented 4 months ago

At inference, we need a scalar value as threshold to identify the input image is ID or OOD. I went through the code and I did not find any function to compute this threshold value or may be I missed something. Can you please tell how to detect a input sample at inference as OOD or not.

Thankyou for your amazing work!

Qinyu-Allen-Zhao commented 1 month ago

Apologies for my late reply!!! I should have enabled notifications for my GitHub repos!

Regarding your question, we follow the conventional practice for evaluating OOD detection, using both FPR95 and AUROC (AUC in our paper).

In our code, please refer to lines 23-35 in utils/metrics.py: https://github.com/Qinyu-Allen-Zhao/OptFSOOD/blob/c09a85e1f15de5a1f45c89419be6a5fff03f88c5/utils/metrics.py#L23-L35

We set recall_level_default=0.95, use sklearn.metrics.roc_curve to compute FPR and TPR at different thresholds, and then obtain the FPR at the smallest TPR >= 0.95. If you'd like, you can print out the thresholds to see the details.

When writing this code, I referenced some of the implementations from OpenOOD, which you might also find useful. Their code is a bit more complex than ours, but you can check it out here https://github.com/Jingkang50/OpenOOD/blob/3d7c483d88f15b514ea051799dde4a0cdce30d9c/openood/evaluators/metrics.py#L29-L36

I hope this helps! 😊

Best, Qinyu