Open glenn-jocher opened 3 years ago
Hello,
We can not reproduce coco api's mAP by provided .txt label files.
Coco api will ignore false positive bboxes which matched ignored criteria.
For convenient to participant, I suggest you to provide additional .txt for bboxes with iscrowd=1
.
By the way, slow speed of coco api is mainly due to it treat ap50, ap55, ..., ap95 as independent cases. While in this repo, the match function is decided by iou50 only.
FYI, there exists some faster solutions which match pycocotools extactly: https://github.com/facebookresearch/detectron2/blob/master/detectron2/evaluation/fast_eval_api.py.
Is the script you use for producing the coco evaluation available?
ππΆ Ultralytics Competition: pycocotools mAP Alignment
I'm super excited to announce our very first Ultralytics AI Competition π!! This is the first in a series of challenges that we are facing at Ultralytics that we want to share with the community to allow everyone to help with. Competition participants can contribute open-source solutions to receive prize money and to improve the YOLOv5 experience for everyone.
Problem π€
Our local mAP results do not align as well as we'd like with pycocotools mAP results. They are similar to within about 1%, but we want to improve this further, to give better confidence in the metrics we produce on custom datasets where pycocotools-format JSON labels are unavailable. Below is an example of the problem from our Colab notebook, showing YOLOv5x COCO local 49.6 mAP@0.5:0.95 vs pycocotools 50.7 mAP@0.5:0.95.
YOLOv5 mAP is computed in val.py with functions from utils/metrics.py: https://github.com/ultralytics/yolov5/blob/f17c86b7f0d2038288d7292cb82dec2433cc91e5/val.py#L237-L243
Solution π‘
The winning solution must meet all of the following criteria:
test.py
execution time must remain within 10% of current time for all test cases. If test.py currently runs in 60 seconds, a winning submission must run in <66 seconds. This will prevent a simple import of the pycocotools code, which itself is very slow, and is the reason we have not adopted it fully.Test cases are below (8 different scenarios). This code can be copied and pasted into a Colab notebook, and the cell output should accompany competition submissions.
πΆ Prize β¬1000.00
The first submission that meets all of the Solution requirements will claim the full prize funds of β¬1000.00 (1000 EUR) from Ultralytics. Funds will be converted to participant's local currency using the exchange rate on date of winning submission.
π Deadline
The deadline for submissions is March 31st, 2022. After this date the competition will be closed. If a winning submission is received before the deadline then the competition may be closed early.
If multiple submissions meet the Solution requirements the earliest dated submission will claim the full prize. If a PR is composed of multiple commits then the submission timestamp will be the time of the earliest commit in a PR that meets all of the Solution requirements.
β Participation Requirements
This competition is open to any individual or organization from any country πΊπ³. There is no restrictions on citizenship, age, gender or location. To receive prize funds the winning participant must meet the eligibility requirements of our funds transfer provider and provide name, address, phone number, email and bank info.