mAP by Darknet and mAP by Cartucho are different on detection same validation images.

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )

http://pjreddie.com/darknet/

Other

21.73k stars 7.96k forks source link

mAP by Darknet and mAP by Cartucho are different on detection same validation images. #7332

Open HanTnx opened 3 years ago

HanTnx commented 3 years ago

Let's me explain shortly what I have done. Firstly, I trained my custom dataset on Darknet Ubuntu 18.04, the best weigtht on 840 validation images is mAP 87%. Secondly, I use this 87% best weight on OpenCV DNN Python code to detect the same 840 images and used the detection results to calculate mAP with Cartucho source code. But these two mAP are very different, you can see in include image mAP_curto

Can you explain or correct what I have done wrongly ? Thank you for your time and consideration

AlexeyAB commented 3 years ago

Detections for mAP should be calculated with low confidence-threshold like -thresh 0.005 so you should use the same threshold in OpenCV-DNN to compare apples with apples https://github.com/AlexeyAB/darknet/blob/64efa721ede91cd8ccc18257f98eeba43b73a6af/src/detector.c#L995
OpenCV-DNN uses letter_box instead of resize by default, while Darknet uses resize by default (for training and detection, except yolov4-csp and yolov4x-mish models) https://github.com/AlexeyAB/darknet/issues/232#issuecomment-336955485
Also what accuracy does calculate Cartucho code, is it mAP50, mAP75 or mAP50...95, how many points in Precision-Recall-curve are used?

HanTnx commented 3 years ago

To answer to your question

Accuracy Cartucho code calculate, it is mAP 50.
Points in Precision-Recal-curve are used, I am not sure from reading his code but he follows VOC challenge 2012 which is the same as Darknet instruction
I follow your instruction to set confidence-threshold in OpenCV-DNN 4.4.0 as follows, and the mAP by Cartucho increase from 67% to 73% mAP50
The final problem, I used Darknet yolov4 models (not yolov4-csp and yolov4x-mish models ), so OpenCV-DNN uses letter_box instead of resize by default. The OpenCV I used is 4.4.0, how about update to 4.5 (or latest one) solve this problem ? or Can you suggest me another framework with Python (pytorch, tensorflow) uses resize by default for detection ? Thank you

poornimajd commented 3 years ago

@HanTnx were you able to solve this difference in map issue?

matt-sharp commented 3 years ago

@HanTnx I'm also curious to know if you found a solution to get a better match between Darknet and Open-CV dnn results.