Yolo-V3 results are matching with original paper results

[Sudhakar17]

I used darknet yolo-v3(alexeyAB) to predict the validation 5000 images from COCO2014 which gives mAP of 54.37%

detections_count = 237764, unique_truth_count = 35757 class_id = 0, name = person, ap = 70.23 % class_id = 1, name = bicycle, ap = 51.34 % class_id = 2, name = car, ap = 59.17 % class_id = 3, name = motorbike, ap = 66.61 % class_id = 4, name = aeroplane, ap = 74.85 % class_id = 5, name = bus, ap = 82.59 % class_id = 6, name = train, ap = 78.15 % class_id = 7, name = truck, ap = 53.73 % class_id = 8, name = boat, ap = 46.05 % class_id = 9, name = traffic light, ap = 49.60 % class_id = 10, name = fire hydrant, ap = 79.95 % class_id = 11, name = stop sign, ap = 75.76 % class_id = 12, name = parking meter, ap = 55.05 % class_id = 13, name = bench, ap = 35.32 % class_id = 14, name = bird, ap = 45.61 % class_id = 15, name = cat, ap = 78.01 % class_id = 16, name = dog, ap = 78.04 % class_id = 17, name = horse, ap = 74.77 % class_id = 18, name = sheep, ap = 56.76 % class_id = 19, name = cow, ap = 54.29 % class_id = 20, name = elephant, ap = 83.35 % class_id = 21, name = bear, ap = 79.79 % class_id = 22, name = zebra, ap = 78.85 % class_id = 23, name = giraffe, ap = 85.39 % class_id = 24, name = backpack, ap = 34.76 % class_id = 25, name = umbrella, ap = 57.99 % class_id = 26, name = handbag, ap = 24.16 % class_id = 27, name = tie, ap = 50.52 % class_id = 28, name = suitcase, ap = 48.95 % class_id = 29, name = frisbee, ap = 74.17 % class_id = 30, name = skis, ap = 39.38 % class_id = 31, name = snowboard, ap = 49.56 % class_id = 32, name = sports ball, ap = 59.74 % class_id = 33, name = kite, ap = 44.62 % class_id = 34, name = baseball bat, ap = 50.73 % class_id = 35, name = baseball glove, ap = 50.52 % class_id = 36, name = skateboard, ap = 68.65 % class_id = 37, name = surfboard, ap = 63.51 % class_id = 38, name = tennis racket, ap = 71.90 % class_id = 39, name = bottle, ap = 44.34 % class_id = 40, name = wine glass, ap = 52.10 % class_id = 41, name = cup, ap = 50.68 % class_id = 42, name = fork, ap = 40.13 % class_id = 43, name = knife, ap = 31.97 % class_id = 44, name = spoon, ap = 28.30 % class_id = 45, name = bowl, ap = 50.64 % class_id = 46, name = banana, ap = 34.18 % class_id = 47, name = apple, ap = 20.15 % class_id = 48, name = sandwich, ap = 51.30 % class_id = 49, name = orange, ap = 34.27 % class_id = 50, name = broccoli, ap = 33.78 % class_id = 51, name = carrot, ap = 25.55 % class_id = 52, name = hot dog, ap = 43.00 % class_id = 53, name = pizza, ap = 59.55 % class_id = 54, name = donut, ap = 45.72 % class_id = 55, name = cake, ap = 50.14 % class_id = 56, name = chair, ap = 44.06 % class_id = 57, name = sofa, ap = 59.58 % class_id = 58, name = pottedplant, ap = 44.44 % class_id = 59, name = bed, ap = 67.93 % class_id = 60, name = diningtable, ap = 46.87 % class_id = 61, name = toilet, ap = 75.49 % class_id = 62, name = tvmonitor, ap = 74.30 % class_id = 63, name = laptop, ap = 70.49 % class_id = 64, name = mouse, ap = 71.63 % class_id = 65, name = remote, ap = 48.55 % class_id = 66, name = keyboard, ap = 67.07 % class_id = 67, name = cell phone, ap = 43.15 % class_id = 68, name = microwave, ap = 70.85 % class_id = 69, name = oven, ap = 51.24 % class_id = 70, name = toaster, ap = 17.49 % class_id = 71, name = sink, ap = 59.61 % class_id = 72, name = refrigerator, ap = 72.01 % class_id = 73, name = book, ap = 17.17 % class_id = 74, name = clock, ap = 72.50 % class_id = 75, name = vase, ap = 51.54 % class_id = 76, name = scissors, ap = 39.51 % class_id = 77, name = teddy bear, ap = 59.71 % class_id = 78, name = hair drier, ap = 9.48 % class_id = 79, name = toothbrush, ap = 36.30 % for thresh = 0.25, precision = 0.61, recall = 0.51, F1-score = 0.56 for thresh = 0.25, TP = 18389, FP = 11799, FN = 17368, average IoU = 48.14 %

mean average precision (mAP) = 0.543651, or 54.37 % Total Detection Time: 227.000000 Seconds

I found your code to add some visualizations for the results. But it gives the mAP of 45.74% only. I didnt change any files in your code. Both AlexeyAB implementation and yours kept the IOU-Threshold of 0.5 but gives different result for AP@50 even in each classes.

[Cartucho]

Yeah, that's strange. Note the toaster for example, while in mine the AP is 2%, in Alexey's is 17.5%... 8 times larger!

Do you think you could zip the mAP folder and send it to my e-mail (to.cartucho@gmail.com)? So that I can debug it

[prateethvnayak]

Is this thing solved ?

I am also getting a difference of 10% on tiny-yolov2. Darknet is doing some magic ? on Darknet i got 56%... and here the same set shows 43.3%

[Cartucho]

Hello, I think it may have to do with the metric.

I used the most recent PASCAL VOC metric (the one from 2012) while if you use the old metric 11-point interpolation you will get slightly different scores.

I am currently implementing unit tests to make sure that the result score is the same as matlab.

[prateethvnayak]

is the changes in thsi function log_average_miss_rate ?

I did have a look and saw that you use 9 evenly spaced points. I tried a small hack and did 11 points. I had a increase of 1% in mAP.

I am guessing there is more to it. Please point me to your reference. I can also take a look. Tks

[cookieKeks]

@prateethvnayak Can you tell me precisely what command you used to get the predictions from yolo?

[prateethvnayak]

@cookieKeks I have my own code for the network-flow in tf, but I inject the darknet weights and cfg. Have written a parser for the .weights file

[varghesealex90]

@Cartucho Any updates on this, I am seeing a 20 % difference.

[Adithia99]

what function to call map in yolo darknet (alexeyAB) ?