search

eccv22-ood-workshop / ROBIN-dataset

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts
http://www.ood-cv.org/
46 stars 6 forks source link

Performance of baseline #3

Open Luffy03 opened 2 years ago

Luffy03 commented 2 years ago

Hello, thanks for your great work first! I run your provided baseline directly but get low performance. The results are quite different with that reported in your paper. Is there sth wrong with the code? 屏幕截图 2022-06-10 171609 屏幕截图 2022-06-10 171720

DTennant commented 2 years ago

Hi, the released dataset is only the validation set, not the full test set we used for the paper. We will open a codalab competition later which will enable people to test their model on the full test set.

Also the code in this repo should be able to reach a 60% or so performance on the validation set that I released using this command:

python3 main.py --data /path/to/download/ROBIN-cls-train/ --val-data /path/to/download/ROBIN-cls-val/ --pretrained
Luffy03 commented 2 years ago

Hi, the released dataset is only the validation set, not the full test set we used for the paper. We will open a codalab competition later which will enable people to test their model on the full test set.

Also the code in this repo should be able to reach a 60% or so performance on the validation set that I released using this command:

python3 main.py --data /path/to/download/ROBIN-cls-train/ --val-data /path/to/download/ROBIN-cls-val/ --pretrained

I do follow this construction but the top-1 acc on val set is very low, as shown in the fig.......

Luffy03 commented 2 years ago

The acc on the training set is normal 屏幕截图 2022-06-10 220139

Luffy03 commented 2 years ago

Would you please present the detailed results of your baseline method on the val set? It will help me a lot! thx!

DTennant commented 2 years ago

Hi, we will update the repo with more information ASAP. For now, we are still preparing the data, please stay tuned.

Luffy03 commented 2 years ago

Hi, we will update the repo with more information ASAP. For now, we are still preparing the data, please stay tuned.

Thank you very much! Looking forward to it.

DTennant commented 2 years ago

Hi, Thanks again for your interest in our workshop, and we apologise for the late update. I have update the README in this repo and update a link to the full test set and labels in the README, the codalab submission server will also be online within this week.

Here are some changes to the competition:

  1. The Phase-1 of the competition will not be a code submission challenge, we have released all the test data and labels in this repo. And Phase-1 will last longer than original planed, we will ask each team to provide a description of their developing environment at the end of Phase-1, Phase-2 will still be code submission challenge.
  2. We will be using Top-1 accuracy for image-classification, mAP@50 for object detection, and Acc@pi/6 for pose estimation as the metric, the IID test performance will also be considered as per request of the sponsor, we will penalize submissions that are significantly different in IID performance with our baseline.
  3. The only limitation now is that the model should only be trained on the given training set and/or the ImageNet-1k dataset, no additional dataset is allowed. You can use any ensemble, data augmentation, or test-time training techniques.

Thanks again for your patience.

Best regards

khawar-islam commented 1 year ago

Hello @DTennant

Could you please share the "full test set you used for the paper." I have compare your results with my results and thus i need full test dataset fro three tasks img cls, obj detec and pose.