search

eccv22-ood-workshop / ROBIN-dataset

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts
http://www.ood-cv.org/
46 stars 6 forks source link

Can we use ImageNet (or extra data) for pre-training in this competition? #1

Open vtddggg opened 2 years ago

vtddggg commented 2 years ago

Thanks for your great datasets for ood evaluation! I cannot find any official notes about if we can use ImageNet (or extra data) for pre-training in ROBIN Challenge. Could you provide some more details ?

DTennant commented 2 years ago

Hi, thanks for your interest. I have updated a QA in the README, and it is not allowed to use any additional data other than the ImageNet-1k and ImageNet-1k pretrained models.

vtddggg commented 2 years ago

Thanks for your response. Apologies for another question: Since ImageNet pretrained models are allowed, can we use any other ensemble or test-time augmentation tricks (without violation of the no-external data rule)

DTennant commented 2 years ago

Hi, Thanks again for your interest in our workshop, and we apologise for the late update. I have update the README in this repo and update a link to the full test set and labels in the README, the codalab submission server will also be online within this week.

Here are some changes to the competition:

  1. The Phase-1 of the competition will not be a code submission challenge, we have released all the test data and labels in this repo. And Phase-1 will last longer than original planed, we will ask each team to provide a description of their developing environment at the end of Phase-1, Phase-2 will still be code submission challenge.
  2. We will be using Top-1 accuracy for image-classification, mAP@50 for object detection, and Acc@pi/6 for pose estimation as the metric, the IID test performance will also be considered as per request of the sponsor, we will penalize submissions that are significantly different in IID performance with our baseline.
  3. The only limitation now is that the model should only be trained on the given training set and/or the ImageNet-1k dataset, no additional dataset is allowed. You can use any ensemble, data augmentation, or test-time training techniques.

Thanks again for your patience.

Best regards