search

Computer-Vision-in-the-Wild / DataDownload

MIT License
24 stars 4 forks source link

Question on experiment settings #2

Closed vincentlux closed 2 years ago

vincentlux commented 2 years ago

Hi, thanks a lot for organizing this benchmark dataset! Can you help clarify two details of the OD benchmark?

  1. For the eval scores reported in the ELEVATER paper, what is the evaluation metric? is it mAP or mAP@50?
  2. For the eval scores reported in the ELEVATER paper, are these scores from the test set or the validation set? Does this line indicate that the number is from the validation set?

Thanks a lot for your help!

liunian-harold-li commented 2 years ago

Hi, thank you for your interest in our benchmark

  1. It's mAP@50:95 as in COCO evaluation.
  2. We used test set for final reporting. The config was for fine-tuning. For evaluation, we changed the TEST to "test" in the eval code.
vincentlux commented 2 years ago

Got it! Thanks for your quick reponse!