search

rafaelpadilla / review_object_detection_metrics

Object Detection Metrics. 14 object detection metrics: mean Average Precision (mAP), Average Recall (AR), Spatio-Temporal Tube Average Precision (STT-AP). This project supports different bounding box formats as in COCO, PASCAL, Imagenet, etc.
Other
1.09k stars 215 forks source link

Incorrect mAP when categories are not used #152

Open spamekkel opened 1 week ago

spamekkel commented 1 week ago

Hi,

Thanks for sharing your code. I have tried this and found a bug in the mAP calculation when the 'categories' in the annotations.json file contain more then used in the annotations. For example, in the tube test annotations.jon I've added 'frog'. Initially, the mAP is 1.0: predictions are 100% correct. But when 'frog' is added to the categories, the mAP is 0.66.

"categories": [ { "id": 0, "name": "cat" }, { "id": 1, "name": "dog" }, { "id": 2, "name": "frog" }

Could you please check your code ?

rafaelpadilla commented 5 days ago

Hi @spamekkel, 😄

Thank you for your comment.

Based on the few details you provided, I believe there is no issue with the code. Mean Average Precision is computed by averaging the Average x Precision over all categories (classes) of your dataset.

Initially you have 2 classes and a mAP of 1. When you add a new class (frog), your results will be dropped by 1/3, since you made no detection in this new class.

Is that the case? I am missing anything here? 🤔 Please, let me know if I understood correctly.

spamekkel commented 3 days ago

When calculating mAP, if a certain class has no ground truth or predictions, it essentially means that the average precision (AP) for that class is undefined.

The class with no ground truth or predictions is typically excluded from the mAP calculation. mAP is the mean of the APs of all classes, so if a class has no data, it doesn’t contribute to the mean. This means the mAP is calculated over the remaining classes that have valid AP values.