search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.77k stars 7.96k forks source link

Strange mAP results on MS COCO even with original weights #6670

Open johnny-mueller opened 4 years ago

johnny-mueller commented 4 years ago

I work with the MS COCO 2017 dataset and currently run the mAP calculation with the evaluation dataset. Furthermore I use the Yolov3.cfg (416x416)

My procedure is as follows:

  1. I download the original weights: 

    wget https://pjreddie.com/media/files/yolov3.weights

  2. I start the calculation with the following command:

    /darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.50 -points 101

  3. And I get a mean average precision (mAP@0.50) = 0.330887, or 33.09

Is there something wrong with my procedure or why is the deviation from the publications so different?

My results look like the following:

Loading weights from yolov3.weights...
 seen 64, trained: 32013 K-images (500 Kilo-batches_64)
Done! Loaded 107 layers from weights-file

 calculation mAP (mean average precision)...
 Detection layer: 82 - type = 28
 Detection layer: 94 - type = 28
 Detection layer: 106 - type = 28
5000
 detections_count = 249373, unique_truth_count = 73562
class_id = 0, name = person, ap = 37.96%         (TP = 7979, FP = 3465)
class_id = 1, name = bicycle, ap = 29.46%        (TP = 156, FP = 64)
class_id = 2, name = car, ap = 33.22%            (TP = 1198, FP = 527)
class_id = 3, name = motorbike, ap = 37.83%      (TP = 238, FP = 56)
class_id = 4, name = aeroplane, ap = 44.36%      (TP = 116, FP = 6)
class_id = 5, name = bus, ap = 44.13%            (TP = 231, FP = 43)
class_id = 6, name = train, ap = 46.08%          (TP = 161, FP = 16)
class_id = 7, name = truck, ap = 33.32%          (TP = 247, FP = 125)
class_id = 8, name = boat, ap = 25.94%           (TP = 179, FP = 83)
class_id = 9, name = traffic light, ap = 28.01%          (TP = 323, FP = 122)
class_id = 10, name = fire hydrant, ap = 43.43%          (TP = 81, FP = 7)
class_id = 11, name = stop sign, ap = 38.88%     (TP = 52, FP = 7)
class_id = 12, name = parking meter, ap = 39.62%         (TP = 44, FP = 10)
class_id = 13, name = bench, ap = 24.57%         (TP = 173, FP = 93)
class_id = 14, name = bird, ap = 26.13%          (TP = 226, FP = 188)
class_id = 15, name = cat, ap = 47.07%           (TP = 184, FP = 25)
class_id = 16, name = dog, ap = 43.36%           (TP = 185, FP = 71)
class_id = 17, name = horse, ap = 42.56%         (TP = 227, FP = 97)
class_id = 18, name = sheep, ap = 36.57%         (TP = 254, FP = 112)
class_id = 19, name = cow, ap = 34.37%           (TP = 240, FP = 99)
class_id = 20, name = elephant, ap = 41.47%      (TP = 183, FP = 40)
class_id = 21, name = bear, ap = 45.03%          (TP = 61, FP = 15)
class_id = 22, name = zebra, ap = 42.89%         (TP = 205, FP = 18)
class_id = 23, name = giraffe, ap = 44.25%       (TP = 192, FP = 13)
class_id = 24, name = backpack, ap = 18.15%      (TP = 113, FP = 87)
class_id = 25, name = umbrella, ap = 35.08%      (TP = 260, FP = 116)
class_id = 26, name = handbag, ap = 17.79%       (TP = 152, FP = 98)
class_id = 27, name = tie, ap = 28.20%           (TP = 140, FP = 74)
class_id = 28, name = suitcase, ap = 30.80%      (TP = 157, FP = 50)
class_id = 29, name = frisbee, ap = 43.69%       (TP = 96, FP = 15)
class_id = 30, name = skis, ap = 23.51%          (TP = 93, FP = 37)
class_id = 31, name = snowboard, ap = 28.92%     (TP = 37, FP = 15)
class_id = 32, name = sports ball, ap = 30.01%           (TP = 154, FP = 79)
class_id = 33, name = kite, ap = 26.67%          (TP = 142, FP = 52)
class_id = 34, name = baseball bat, ap = 32.76%          (TP = 86, FP = 34)
class_id = 35, name = baseball glove, ap = 35.75%        (TP = 95, FP = 20)
class_id = 36, name = skateboard, ap = 40.51%            (TP = 134, FP = 22)
class_id = 37, name = surfboard, ap = 35.79%     (TP = 182, FP = 61)
class_id = 38, name = tennis racket, ap = 41.42%         (TP = 170, FP = 23)
class_id = 39, name = bottle, ap = 28.88%        (TP = 544, FP = 318)
class_id = 40, name = wine glass, ap = 30.27%            (TP = 189, FP = 89)
class_id = 41, name = cup, ap = 32.87%           (TP = 531, FP = 259)
class_id = 42, name = fork, ap = 28.31%          (TP = 107, FP = 58)
class_id = 43, name = knife, ap = 18.87%         (TP = 109, FP = 86)
class_id = 44, name = spoon, ap = 18.04%         (TP = 84, FP = 70)
class_id = 45, name = bowl, ap = 33.59%          (TP = 381, FP = 181)
class_id = 46, name = banana, ap = 25.97%        (TP = 157, FP = 76)
class_id = 47, name = apple, ap = 20.18%         (TP = 101, FP = 128)
class_id = 48, name = sandwich, ap = 34.94%      (TP = 101, FP = 37)
class_id = 49, name = orange, ap = 27.21%        (TP = 145, FP = 85)
class_id = 50, name = broccoli, ap = 22.71%      (TP = 122, FP = 75)
class_id = 51, name = carrot, ap = 20.26%        (TP = 133, FP = 117)
class_id = 52, name = hot dog, ap = 30.27%       (TP = 65, FP = 41)
class_id = 53, name = pizza, ap = 38.67%         (TP = 198, FP = 47)
class_id = 54, name = donut, ap = 31.76%         (TP = 227, FP = 185)
class_id = 55, name = cake, ap = 35.36%          (TP = 199, FP = 78)
class_id = 56, name = chair, ap = 28.72%         (TP = 918, FP = 514)
class_id = 57, name = sofa, ap = 37.41%          (TP = 183, FP = 88)
class_id = 58, name = pottedplant, ap = 30.57%           (TP = 175, FP = 82)
class_id = 59, name = bed, ap = 40.02%           (TP = 124, FP = 43)
class_id = 60, name = diningtable, ap = 28.16%           (TP = 364, FP = 240)
class_id = 61, name = toilet, ap = 43.29%        (TP = 138, FP = 10)
class_id = 62, name = tvmonitor, ap = 43.29%     (TP = 229, FP = 51)
class_id = 63, name = laptop, ap = 41.40%        (TP = 172, FP = 38)
class_id = 64, name = mouse, ap = 42.15%         (TP = 86, FP = 23)
class_id = 65, name = remote, ap = 29.77%        (TP = 144, FP = 62)
class_id = 66, name = keyboard, ap = 40.22%      (TP = 103, FP = 19)
class_id = 67, name = cell phone, ap = 30.95%            (TP = 150, FP = 65)
class_id = 68, name = microwave, ap = 43.14%     (TP = 45, FP = 12)
class_id = 69, name = oven, ap = 34.26%          (TP = 86, FP = 30)
class_id = 70, name = toaster, ap = 19.24%       (TP = 2, FP = 0)
class_id = 71, name = sink, ap = 35.75%          (TP = 140, FP = 50)
class_id = 72, name = refrigerator, ap = 39.60%          (TP = 85, FP = 25)
class_id = 73, name = book, ap = 13.02%          (TP = 409, FP = 980)
class_id = 74, name = clock, ap = 40.10%         (TP = 192, FP = 29)
class_id = 75, name = vase, ap = 31.80%          (TP = 160, FP = 80)
class_id = 76, name = scissors, ap = 26.84%      (TP = 17, FP = 3)
class_id = 77, name = teddy bear, ap = 34.67%            (TP = 110, FP = 23)
class_id = 78, name = hair drier, ap = 11.11%            (TP = 0, FP = 0)
class_id = 79, name = toothbrush, ap = 23.94%            (TP = 26, FP = 9)

 for conf_thresh = 0.25, precision = 0.68, recall = 0.31, F1-score = 0.42
 for conf_thresh = 0.25, TP = 22597, FP = 10491, FN = 50965, average IoU = 53.57 %

 IoU threshold = 50 %, used 101 Recall-points
 mean average precision (mAP@0.50) = 0.330887, or 33.09 %
Total Detection Time: 123 Seconds
marina-neseem commented 4 years ago

Same problem! Did you figure out what's wrong?