search

eccv22-ood-workshop / ROBIN-dataset

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts
http://www.ood-cv.org/
46 stars 6 forks source link

The categories of the detection track #4

Closed zhengye1995 closed 2 years ago

zhengye1995 commented 2 years ago

Thanks for your great benchmark for ood evaluation.

After I unzip the dataset for detection track, I found that there are 20 categories in the annotation of the dataset instead of the 10 in the workshop's homepage. What are the actual categories used for detection track? If there are only 10 categories, are images containing other categories used as backgrounds or I should drop them?

DTennant commented 2 years ago

Hi, we will update the repo with more information ASAP. For now, we are still preparing the data, please stay tuned.

DTennant commented 2 years ago

Hi, Thanks again for your interest in our workshop, and we apologise for the late update. I have update the README in this repo and update a link to the full test set and labels in the README, the codalab submission server will also be online within this week.

Here are some changes to the competition:

  1. The Phase-1 of the competition will not be a code submission challenge, we have released all the test data and labels in this repo. And Phase-1 will last longer than original planed, we will ask each team to provide a description of their developing environment at the end of Phase-1, Phase-2 will still be code submission challenge.
  2. We will be using Top-1 accuracy for image-classification, mAP@50 for object detection, and Acc@pi/6 for pose estimation as the metric, the IID test performance will also be considered as per request of the sponsor, we will penalize submissions that are significantly different in IID performance with our baseline.
  3. The only limitation now is that the model should only be trained on the given training set and/or the ImageNet-1k dataset, no additional dataset is allowed. You can use any ensemble, data augmentation, or test-time training techniques.

Thanks again for your patience.

Best regards