calculate mAP@0.50 using -map got different result as using pycocotools

tino926 commented 3 years ago

I got different mAP@0.50 values by using darknet's "map" option and by using pycocotools:

I first use

      ./darknet detector map \
      test_data/mydata.data \
      run/yolov4-tiny-custom.cfg  \
      run/run3/yolov4-tiny-custom_best.weights

got mAP@0.5 = 0.82:

 detections_count = 30104, unique_truth_count = 724
class_id = 0, name = person, ap = 82.34%         (TP = 583, FP = 581)
... ... 
 mean average precision (mAP@0.50) = 0.823401, or 82.34 %

Then i use

./darknet detector valid \
test_data/mydata.data run/yolov4-tiny-custom.cfg \
run/run3/yolov4-tiny-custom_best.weights

and pycocotools, got mAP@0.5 = 0.803:

>>> cocoEval.summarize()
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.503
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.803
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.591
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.057
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.511
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.399
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.638
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.658
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.164
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.666

The two mAP values show significant difference. Another strange thing is: "darknet detector map" shows that "detections_count = 30104", however, in my coco_result.json from "darknet detector valid" shows 123952 bounding boxes.

Did I do something wrong?

lsd1994 commented 3 years ago

darknet only use detections with prob>0.005 to calculate map, I don't know how about pycocotools, you can check it.

AlexeyAB commented 3 years ago

Its strange that you get negative Precision and Recall values for small objects in pycocotools.

Fushier commented 3 years ago