search

eccv22-ood-workshop / ROBIN-dataset

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts
http://www.ood-cv.org/
46 stars 6 forks source link

Low Quality Validation Set #9

Closed JeremyCJM closed 2 years ago

JeremyCJM commented 2 years ago

Hi all,

I compared your new classification dataset (v1.0) with the previous version recently. The quality of the current validation set is pretty lower -- and there are many unrelated objects appearing in the images, e.g. a "car" image has a bike in the foreground and a car in the background, a "car" image with a very large airplane and a very small car. I believe those images are not appropriate to evaluate the new OOD issues proposed in your paper.

I am wondering if there is any mistake in the data collection that you may not notice? And would this v1.0 be the final dataset we use for the challenge?

Thanks, Jeremy

mmmmimic commented 2 years ago

Hi Jeremy,

Could you let me know if you have addressed the problem? I think I have the same question with you:)

Thanks! Manxi

JeremyCJM commented 2 years ago

It seems that v1.0 is for all three tasks, and the author provided the processing scripts to extract data from v1.0. But I sincerely suggest the author provide the processed data so that everyone can have exactly the same data.

mmmmimic commented 2 years ago

Thanks! How could I access the processing scripts? I tried this but failed to find the file.