search

eccv22-ood-workshop / ROBIN-dataset

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts
http://www.ood-cv.org/
46 stars 6 forks source link

dataset and baseline about object detection? #12

Open Ice-cool opened 2 years ago

Ice-cool commented 2 years ago
  1. Dataset: In the Q&A, the ImageNet-1k is allowed in the image classification. What about the boxes-level annotations? (https://image-net.org/download-bboxes.php) 
    
Q: Is it allowed to use additional data for training purposes?

A: No, only ImageNet-1k and ImageNet-1k pretrained models can be used for the challenge.



2. Baseline:
in the paper, the iid-test (AP50)=72%, however, only 60-mAP in our re-implement with the same setting as the appendix mentioned. Is this the matter about dataset or other things?
DTennant commented 2 years ago

No, Only the images and the classification labels from ImageNet-1k can be used. I will update the QA accordingly.

Ice-cool commented 2 years ago

@DTennant Thank you. When will the baseline about object detection release?

DTennant commented 2 years ago

Hi, We are still waiting for the confirmation from our sponsor, the baseline for the three tasks as well as the data processing script and detailed evaluation protocol will be released after everything is confirmed.

Ice-cool commented 2 years ago

The annotations are incomplete. For example, there are three chairs in iid_test/Images/chair/2008_007194.jpg, but only two chairs are annotated. (there are many similar images in test data)

DTennant commented 2 years ago

Hi, We have released the codalab server for evaluation, for the detection baseline, we aim to release it by the weekend.