search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.76k stars 7.96k forks source link

Can't Reproduce the paper mAP #3120

Open prateethvnayak opened 5 years ago

prateethvnayak commented 5 years ago

@AlexeyAB .. Is it True that using Tensorflow implementation, the best mAP you can reach is ~44% ? using the darknet weights and cfg file. and implementing the network in TF, just not able to reproduce the paper accuracy. All the things checked -

Is anyone able to reproduce this accuracy in Tensorflow ? I appreciate if any of have insights on this.

AlexeyAB commented 5 years ago

I didn't try to implement Yolo on TF, but using PyTorch (Caffe2) you can get the same mAP on MS COCO dataset as on the Darknet for Yolo v3: https://github.com/ultralytics/yolov3#darknet-conversion

prateethvnayak commented 5 years ago 
* Use `nms_threshold = 0.45` and `probability_thresh = 0.001` for mAP calculation.

Okay I will try these values and update the reply here. 

* Did you train and detect by using TF, or just detect by using TF?

No, this is Inference only code. Using the pre-trained Darknet weights and cfg file

* Did you train by using PascalVOC 2007+2012?

If I am not wrong, the pre-trained model is trained on PascalVOC 2007+2012 (train+val) set. I am using just the VOC2007 test set to reproduce as mentioned.

* Do you use yolo-voc.cfg or yolov3-voc.cfg?

I haven't gone to yolov3. All the models I tested were yolov2 versions. I got the following results tiny-yolo-voc.cfg ---> 44 mAP (claimed 57) tiny-yolo.cfg ----> 17 mAP (no claimed mAP) Yolov2 (voc) ---> 55 mAP (claimed 76) I used the corresponding weights from the same website

* Did you implement Yolo model on TF correctly?

Yes. This has been thoroughly checked and compared with many implementations online, as well as some test images which gave same output as Darknet. So the loading of weights is correct for the TF kernels.

I didn't try to implement Yolo on TF, but using PyTorch (Caffe2) you can get the same mAP on MS COCO dataset as on the Darknet for Yolo v3: https://github.com/ultralytics/yolov3#darknet-conversion

Okay. I shall give this a try. Haven't tried with PyTorch yet. There is no claim anywhere of reproducibility of Darknet accuracy on any other framework. Please point me a reference if you see one. I did use Darkflow, but it has a lot of bugs in weight loader and inference.

AlexeyAB commented 5 years ago

Also you should know that this cfg-file https://github.com/thtrieu/darkflow/blob/master/cfg/yolo-voc.cfg and this https://github.com/pjreddie/darknet/blob/master/cfg/yolov2-voc.cfg are different.

Do you use Python script for testing? reval_voc_py3.py --year 2007 --classes data\voc.names --image_set test --voc_dir E:\VOC2007_2012\VOCtrainval_11-May-2012\VOCdevkit results

prateethvnayak commented 5 years ago

@AlexeyAB Yes. I am aware of that.

A quick update on the results I got just now. I ran the Yolov2 (608x608) which is on COCO dataset with corresponding weights. And I got a mAP - 42 which is pretty close to the actual claimed mAP - 48.

Do you use Python script for testing? reval_voc_py3.py --year 2007 --classes data\voc.names --image_set test --voc_dir E:\VOC2007_2012\VOCtrainval_11-May-2012\VOCdevkit results

I shall try this and let you know

So, I'm confident about my implementation. Do you think the mAP code is not correct ? I am using this one - mAP code

AlexeyAB commented 5 years ago

I ran the Yolov2 (608x608) which is on COCO dataset with corresponding weights. And I got a mAP - 42 which is pretty close to the actual claimed mAP - 48.

Did you get it by using pycocotool or by using https://github.com/Cartucho/mAP ?

Do you think the mAP code is not correct ?

I think yes.

Use reval_voc_py3.py For example, as it is used with Darknet: https://github.com/AlexeyAB/darknet/blob/master/build/darknet/x64/calc_mAP_voc_py.cmd