search

thtrieu / darkflow

Translate darknet to tensorflow. Load trained weights, retrain/fine-tune using tensorflow, export constant graph def to mobile devices
GNU General Public License v3.0
6.13k stars 2.08k forks source link

mAP is low compared to YOLO #957

Open aswin-raghavan opened 5 years ago

aswin-raghavan commented 5 years ago

Thanks for darkflow. Clearly a lot of effort was put in to this.

I trained yolo.cfg on VOC2007 data using ADAM for > 100k steps, and eventually the loss was around 2.0xxx. I am aware this is not as good as the darknet loss of < 1.0, but darkflow seems to implement a slightly different loss function wrt rescore.

I also realized that you may have wrongly linked the testing dataset of VOC2007 as the training dataset, but this might not have any major impact on the result.

This trained model has a significantly lower mAP of 35%, compared to YOLO, and I wonder why? What can I do to improve this?

[mAP was calculated using https://github.com/rafaelpadilla/Object-Detection-Metrics#how-to-use-this-project]


AP: 30.83% (bicycle)
AP: 31.67% (bird)
AP: 23.59% (boat)
AP: 19.62% (bottle)
AP: 38.62% (bus)
AP: 47.79% (car)
AP: 50.24% (cat)
AP: 18.99% (chair)
AP: 31.74% (cow)
AP: 20.83% (diningtable)
AP: 40.26% (dog)
AP: 54.32% (horse)
AP: 40.22% (motorbike)
AP: 42.81% (person)
AP: 12.18% (pottedplant)
AP: 26.44% (sheep)
AP: 31.54% (sofa)
AP: 49.66% (train)
AP: 39.75% (tvmonitor)
mAP: 35.12%```
aswin-raghavan commented 5 years ago

Follow up: I realize that this might not have anything to do with training as in darkflow. I pulled the tiny yolov2-voc cfg and weights from here (Table in https://pjreddie.com/darknet/yolov2/) where they claim an mAP of 57% but instead what I get is an mAP of 26%. So either

  1. The darkflow code that loads the cfg and weights has a bug leading to low accuracy.
  2. The evaluation script is different than what is claimed in darknet/yolo.
Alemolet commented 5 years ago

Any other clue about why this happens? I noticed a really low mAP on my results, too, but I thought it was because those pre-trained weights don't really fit good my research case (pedestrian detection from drones, so really tiny and small subjects to detect, as well as a lot of occlusion, bad quality and camera angle very different from the traditional pedestrian detection training images)

UsernameIsName-bot commented 4 years ago

@Alemolet I cant say im well versed in YOLO but im pretty sure its performance dips when it comes to small objects. I think SSD would be a better alternative.

suprito commented 4 years ago

Can anyone tell me how to find the mAP value for yolo v2 darkflow? I am using windows system