search

eccv22-ood-workshop / ROBIN-dataset

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts
http://www.ood-cv.org/
46 stars 6 forks source link

When will the colab evaluation server be released? #8

Open Chan-Sun opened 2 years ago

Chan-Sun commented 2 years ago

Hi! When will the colab evaluation server be released? thx

DTennant commented 2 years ago

Hi, Thanks for your interest in our workshop. We are checking the final evaluation details with our sponsor, and the server will be online after everything is confirmed. I expect this to be within this week.

DTennant commented 2 years ago

Hi, Thanks again for your interest in our workshop, and we apologise for the late update. I have update the README in this repo and update a link to the full test set and labels in the README, the codalab submission server will also be online within this week.

Here are some changes to the competition:

  1. The Phase-1 of the competition will not be a code submission challenge, we have released all the test data and labels in this repo. And Phase-1 will last longer than original planed, we will ask each team to provide a description of their developing environment at the end of Phase-1, Phase-2 will still be code submission challenge.
  2. We will be using Top-1 accuracy for image-classification, mAP@50 for object detection, and Acc@pi/6 for pose estimation as the metric, the IID test performance will also be considered as per request of the sponsor, we will penalize submissions that are significantly different in IID performance with our baseline.
  3. The only limitation now is that the model should only be trained on the given training set and/or the ImageNet-1k dataset, no additional dataset is allowed. You can use any ensemble, data augmentation, or test-time training techniques. Thanks again for your patience.

Best regards

Chan-Sun commented 2 years ago

Thanks for your work! Quite look forward to this competition!

I notice there is a folder named iid_test besides train folder, so does that mean these iid data are the test data for Phase-1 and ood will appear in Phase-2?

DTennant commented 2 years ago

no there is folder named nuisance that is the ood test data. We will be using the iid test data to test the iid performance which will be used to determine whether a submission is valid or not, details will be released after we have confirmed this with our sponsor.

Chan-Sun commented 2 years ago

that's clear, thanks.

luan-g commented 2 years ago

@DTennant Hi, It seems that the new dataset is different with the previous one. And the numbers of images in the new dataset is different with the numbers in the paper. What’s changed？

DTennant commented 2 years ago

Hi, The codalab evaluation server is released, you can checkout the README to find the link to the server