search

Jingkang50 / OpenOOD

Benchmarking Generalized Out-of-Distribution Detection
MIT License
858 stars 108 forks source link

Copy/paste error in CIFAR 10 OoD configuration? #140

Closed romain-xu-darme closed 1 year ago

romain-xu-darme commented 1 year ago

Hello, thanks a lot for developing this library! It seems that there is an issue in the OoD configuration for CIFAR10: in configs/datasets/cifar10/cifar10_ood.yml, shouldn't the validation set point to cifar10 rather than cifar100? Thanks

Jingkang50 commented 1 year ago

The validation set for CIFAR-10 as ID have 2 parts: CIFAR-10's val as ID and CIFAR-100's val as OOD. The goal is to find the optimal hyperparam for OOD detection, so we have validation data from both ID and OOD set.

romain-xu-darme commented 1 year ago

Thank you. Now I understand ... but then why is CIFAR100 also using val_cifar100 as OoD validation set in cifar100_ood.yml ?

Jingkang50 commented 1 year ago

You are correct. cifar100_ood should use val_cifar10.txt I will double-check with my teammates on this issue. Thank you for pointing it out. Nevertheless, the problem shall not affect too much on our main results and conclusion, since if different hyperparam makes no difference on val, we will take default hyperparam for the result. Anyhow, we will look into this issue. Thanks!

romain-xu-darme commented 1 year ago

You're welcome :)