search

eccv22-ood-workshop / ROBIN-dataset

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts
http://www.ood-cv.org/
46 stars 6 forks source link

About IID Threshold #25

Open HanshengGUO opened 2 years ago

HanshengGUO commented 2 years ago

Hi,

Thank you for your careful preparation on this challenge. I’m on the object detection track and worrying the gap between initial test set and final test set since there is a IID threshold requirement. Are the I.I.D. accuracy thresholds produced on the final test set or initial test set? Since the threshold is 79.9 mAP50 but I can only get around 65 on the initial test set, I’m worrying if I’m always working on initial test set and when final test set released, my model passed that threshold. The phase 2 is just 5 days. Can you give more information on the gap, or release the code of your baseline?

Thanks again for your wonderful work.

HanshengGUO commented 2 years ago

Can you maybe release the final IID test set earlier? Or are they just the same one?

DTennant commented 2 years ago

the final iid test set will be the same