search

Jingkang50 / OpenOOD

Benchmarking Generalized Out-of-Distribution Detection
MIT License
858 stars 108 forks source link

Model selection #190

Closed ivan-chai closed 4 months ago

ivan-chai commented 1 year ago

Hi!

It's not clear from the paper whether the best or last training checkpoint is used for testing. What is the default choice for both small datasets and ImageNet?

zjysteven commented 1 year ago

For most training it was selected according to the validation ID accuracy. https://github.com/Jingkang50/OpenOOD/blob/989c41dd3e3ebe85e6f5a5702acb6ce555715735/openood/recorders/base_recorder.py#L40

For certain methods the selection criterion could be different (e.g., OpenGAN explicitly needs to look at val OOD AUROC). You can refer to the code under https://github.com/Jingkang50/OpenOOD/tree/main/openood/recorders for details.