How to get mAP on COCO 2014 test dataset, not on validation set?

[chyomin06]

How to get mAP on COCO 2014 test dataset, not on validation set? I couldn't find any upload server for the 2014 COCO test set.

Thanks.

[AlexeyAB]

@hyomin06 Hi,

Test annotations aren't public, so that you could not adjust your model only to the test data and win the competition and take the first place with a bad detection algorithm.

• How to test Darknet on the MSCOCO test dataset: https://github.com/AlexeyAB/darknet/issues/1052#issuecomment-399372477

• https://stackoverflow.com/questions/48368992/how-can-i-use-ms-coco-test-dev-dataset-for-instance-segmentation?answertab=active#tab-top

The test data is just that, test data. They did not release annotations for it. Instead, you train on the train/val datasets and submit results for the test data to the evaluation server. This ensures a fair comparison between different methods.

---

To evaluate successfully results of YOLOv3 on MS COCO test-dev 2014 with https://worksheets.codalab.org

1. Download and un-pack: http://images.cocodataset.org/zips/test2014.zip
2. Set correct paths valid=test.txt in: https://github.com/AlexeyAB/darknet/blob/master/cfg/coco.data
3. Download: https://pjreddie.com/media/files/yolov3.weights
4. Generate a result file *.json file suiting for coco eval ./darknet detector valid cfg/coco.data cfg/yolov3.cfg yolov3.weights
5. rename and zip the result file according to COCO eval
6. select Paricipate --> test-dev , submit the zipped file to https://worksheets.codalab.org
7. refresh and view the eval results

---

To test mAP@0.5 on MS COCO Validation 2014 dataset just:

1. run this script on Linux: https://github.com/AlexeyAB/darknet/blob/master/scripts/get_coco_dataset.sh
2. set correct path in: https://github.com/AlexeyAB/darknet/blob/master/cfg/coco.data
3. download: https://pjreddie.com/media/files/yolov3.weights

4. run command ./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights

   Or instead you can try to use this way by using https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools and this script https://github.com/ydixon/mAP_eval/blob/master/pred_yolo2json.py as written here: https://github.com/AlexeyAB/darknet/issues/2140

[AlexeyAB]

How to get mAP for IoU=0.50, IoU=0.75 and IoU=0.50:.05:.0.95

• Or use MS COCO evaluation server: https://worksheets.codalab.org/ More about it: https://github.com/AlexeyAB/darknet/issues/2145#issuecomment-451583623

• Or use ./darknet detector valid cfg/coco.data cfg/yolov3.cfg yolov3.weights... and use results with https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools

• Or use ./darknet detector test cfg/coco.data cfg/yolov3.cfg yolov3.weights -dont_show -ext_output < /home/dickson/data/coco/5k.txt > result.txt and use results with https://github.com/ydixon/mAP_eval/blob/master/pred_yolo2json.py and then with https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools More about it: https://github.com/AlexeyAB/darknet/issues/2140

• Or use commands ./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.50 -points 101 ... to get mAP@IoU=0.50, mAP@IoU=0.75 and mAP@IoU=0.50:.05:.0.95

mAP@IoU=0.50: ./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.50 -points 101

mAP@IoU=0.75: ./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.75 -points 101

---

To calculate AP@[.5, .95] or the same mAP@IoU=0.50:.05:.0.95. Run many commands and remember mAPs:

./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.50 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.55 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.60 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.65 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.70 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.75 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.80 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.85 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.90 -points 101
./darknet detector map cfg/coco.data cfg/yolov3.cfg yolov3.weights -iou_thresh 0.95 -points 101

Then calculate: AP@[.5, .95] = mAP@IoU=0.50:.05:.0.95 = (mAP@IoU=0.50 + mAP@IoU=0.55 + mAP@IoU=0.60 + mAP@IoU=0.65 + mAP@IoU=0.70 + mAP@IoU=0.75 + mAP@IoU=0.80 + mAP@IoU=0.85 + mAP@IoU=0.90 + mAP@IoU=0.95) / 10

[YeShangyuan]

@AlexeyAB ./darknet detector map ~/darknet/cfg/coco_4classes.data ~/darknet/cfg/yolov3_coco_darknet49_4classes_test.cfg /export3/media/Dataset/ODdata/mscoco/convertVOC_4/backup/yolov3_coco_darknet49_4classes_310000.weights -iou_thresh 0.50 -points 101

4992Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000580608.txt Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000580693.txt Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000580720.txt Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000580870.txt 4996Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000580975.txt Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000581332.txt Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000581593.txt Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000581655.txt 5000Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000581731.txt Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000581781.txt Can't open label file. (This can be normal only if you use MSCOCO): /export3/media/Dataset/ODdata/mscoco/labels/val2014/COCO_val2014_000000581887.txt

detections_count = 78329, unique_truth_count = 0 class_id = 0, name = person, 9 ap = 0.00 % class_id = 1, name = car, ap = 0.00 % class_id = 2, name = cat, ap = 0.00 % class_id = 3, name = dog, ap = 0.00 % for thresh = 0.25, precision = 0.00, recall = -nan, F1-score = -nan for thresh = 0.25, TP = 0, FP = 12796, FN = 0, average IoU = 0.00 %

IoU threshold = 50 % mean average precision (mAP@0.50) = 0.000000, or 0.00 % Total Detection Time: 24.000000 Seconds

[AlexeyAB]

@YeShangyuan

detections_count = 78329, unique_truth_count = 0

You don't have any labels. Your created incorrect dataset.