Evaluate custom detection with COCO evaluation

[sammilei]

I am trying to evaluate my custom detection with YOLO v3-tiny on the COCO eval standard:

I think the approach is to use coco api: then I need to get the COCO annotation + prediction result

1. Does any scrip convert YOLO labels to COCO annotation?

2. To get the prediction result, I run

./darknet detector test train.data model.cfg my.weights -ext_output -dont_show -out result.json < ../list_of_image_location.txt but screen shows:

layer     filters    size              input                output
   0 conv     16  3 x 3 / 1   416 x 416 x   3   ->   416 x 416 x  16 0.150 BF
   1 max          2 x 2 / 2   416 x 416 x  16   ->   208 x 208 x  16 0.003 BF
   2 conv     32  3 x 3 / 1   208 x 208 x  16   ->   208 x 208 x  32 0.399 BF
   3 max          2 x 2 / 2   208 x 208 x  32   ->   104 x 104 x  32 0.001 BF
   4 conv     64  3 x 3 / 1   104 x 104 x  32   ->   104 x 104 x  64 0.399 BF
   5 max          2 x 2 / 2   104 x 104 x  64   ->    52 x  52 x  64 0.001 BF
   6 conv    128  3 x 3 / 1    52 x  52 x  64   ->    52 x  52 x 128 0.399 BF
   7 max          2 x 2 / 2    52 x  52 x 128   ->    26 x  26 x 128 0.000 BF
   8 conv    256  3 x 3 / 1    26 x  26 x 128   ->    26 x  26 x 256 0.399 BF
   9 max          2 x 2 / 2    26 x  26 x 256   ->    13 x  13 x 256 0.000 BF
  10 conv    512  3 x 3 / 1    13 x  13 x 256   ->    13 x  13 x 512 0.399 BF
  11 max          2 x 2 / 1    13 x  13 x 512   ->    13 x  13 x 512 0.000 BF
  12 conv   1024  3 x 3 / 1    13 x  13 x 512   ->    13 x  13 x1024 1.595 BF
  13 conv    256  1 x 1 / 1    13 x  13 x1024   ->    13 x  13 x 256 0.089 BF
  14 conv    512  3 x 3 / 1    13 x  13 x 256   ->    13 x  13 x 512 0.399 BF
  15 conv     30  1 x 1 / 1    13 x  13 x 512   ->    13 x  13 x  30 0.005 BF
  16 yolo
  17 route  13
  18 conv    128  1 x 1 / 1    13 x  13 x 256   ->    13 x  13 x 128 0.011 BF
  19 upsample            2x    13 x  13 x 128   ->    26 x  26 x 128
  20 route  19 8
  21 conv    256  3 x 3 / 1    26 x  26 x 384   ->    26 x  26 x 256 1.196 BF
  22 conv     30  1 x 1 / 1    26 x  26 x 256   ->    26 x  26 x  30 0.010 BF
  23 yolo
Total BFLOPS 5.454 
 Allocate additional workspace_size = 52.43 MB 
Loading weights from my.weights...
 seen 64 
Done!
Enter Image Path: 52.jpg: Predicted in 6.329000 milli-seconds.
backpack: 100%  (left_x:  188   top_y:   22   width:  107   height:  136)
Segmentation fault (core dumped)

and the result.json is empty

I tried

./darknet detector valid train.data model.cfg my.weights -ext_output -dont_show -out result.json < list_of_images.txt the result has all the same image_id as "image_id":0.

Anyone can help? Thank you so much in advance!

[AlexeyAB]

@sammilei Hi,

Read this issue: https://github.com/AlexeyAB/darknet/issues/2145

---

Try to use this command: ./darknet detector valid train.data model.cfg my.weights

With this line: https://github.com/AlexeyAB/darknet/blob/cce34712f6928495f1fbc5d69332162fc23491b9/cfg/coco.data#L7 in the train.data file

[sammilei]

Thank you so much for the quick response!

1. I read that issue before creating this one. It seems all others are talking about evaluating on COCO data, not custom data set. Did I misunderstand it?

sure I can get the ap:[0.5, 0.05, 0.95] by using map option and do the math after but I also need AP across scales, AR/mAR and AR across scales.

2. yes. I put the eval = coco in .data file and it generates the result in coco_results.json. with all image_id = 0. {"image_id":0, "category_id":5, "bbox":[377.408203, 102.690903, 30.087219, 26.766144], "score":0.034358}, {"image_id":0, "category_id":5, "bbox":[383.043427, 101.953255, 14.616455, 23.502960], "score":0.007570}, {"image_id":0, "category_id":1, "bbox":[184.751907, 67.200760, 15.139221, 17.591583], "score":0.012038}, {"image_id":0, "category_id":1, "bbox":[251.860611, 98.421112, 47.097733, 61.968231], "score":0.032457}, {"image_id":0, "category_id":4, "bbox":[251.860611, 98.421112, 47.097733, 61.968231], "score":0.006651}, {"image_id":0, "category_id":1, "bbox":[200.820053, 112.990402, 35.859680, 23.019806], "score":0.169586}, {"image_id":0, "category_id":5, "bbox":[200.820053, 112.990402, 35.859680, 23.019806], "score":0.010340}, {"image_id":0, "category_id":1, "bbox":[185.498505, 104.822067, 42.898041, 40.651520], "score":0.102428},

[AlexeyAB]

I read that issue before creating this one. It seems all others are talking about evaluating on COCO data, not custom data set. Did I misunderstand it?

• Do you want to check all APs on custom dataset?
• Do you have your dataset in MS COCO format?
• Did you try to use pycocotools, and what parameters did you use? https://github.com/cocodataset/cocoapi/tree/master/PythonAPI/pycocotools

[sammilei]

Do you want to check all APs on custom dataset?

• yes. APs, ARs.

Do you have your dataset in MS COCO format?

• no. I have it in YOLO version

  Did you try to use pycocotools, and what parameters did you use?

• no. I only tried to use pycocoDemo.ipynb: https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb (assumed this is to evaluate custom dataset) but I don't have the COCO format of labels and ./darknent valid ... didn't return correct coco_results.json to me. So I am stuck and seeking help here. I am looking into modifying the validate_detector_map() but just worried if making any mistake will generate a wrong evaluation.

[AlexeyAB]

but I don't have the COCO format of labels and ./darknent valid ... didn't return correct coco_results.json to me.

What is the correct coco_results.json for you? What does it mean?

If you want to use cocoapi (pycocoEvalDemo.ipynb / pycocotools) then you should have your Dataset in MS COCO format.

Otherwise you can only use ./darknet detector map...

[sammilei]

What is the correct coco_results.json for you? What does it mean?

fixed the problem by changing static int get_coco_image_id(char *filename) { char *p = strrchr(filename, '/'); char *c = strrchr(filename, '_'); //changed _ to e for my image names as frame0000.jpg if (c) p = c; return atoi(p + 1); }

If you want to use cocoapi (pycocoEvalDemo.ipynb / pycocotools) then you should have your Dataset in MS COCO format.

do you know any script to convert YOLO format to COCO format?

[AlexeyAB]

do you know any script to convert YOLO format to COCO format?

No.

[LukeAI]

This script converts Yolo/COCO and other formats: https://github.com/ssaru/convert2Yolo

[QingxinWx]

Hi，how do you solve this problem “Evaluate custom detection with COCO evaluation”. I think my question is the same as yours, that is, the standard for dividing custom data set into COCO data set

[Fushier]

Hi，how do you solve this problem “Evaluate custom detection with COCO evaluation”. I think my question is the same as yours, that is, the standard for dividing custom data set into COCO data set

You can use this python script to convert darknet format annotations to COCO format. Then use ./darknet detector valid ... to get coco_results.json. Finally use cocopai to calculate.

import pickle
import json
import numpy as np
import cv2
import os

DATA_PATH = '/workspace/data'

def _darknet_box_to_coco_bbox(box, img_width, img_height):
  # darknet  box format: normalized (x_ctr, y_ctr, w, h)
  # coco box format: unnormalized (xmin, ymin, width, height)

  box_width = round(box[2] * img_width, 2)
  box_height = round(box[3] * img_height, 2)
  box_x_ctr = box[0] * img_width
  box_y_ctr = box[1] * img_height

  xmin = round(box_x_ctr - box_width / 2., 2)
  ymin = round(box_y_ctr - box_height / 2., 2)

  bbox = np.array([xmin, ymin, box_width, box_height], dtype=np.float32)
  return [xmin, ymin, box_width, box_height]

if __name__=='__main__':
    cats = list()
    with open('/workspace/data/obj.names', 'r') as f:
        for line in f.readlines():
            line = line.strip('\n')
            cats.append(line)
    cat_info = []
    for i, cat in enumerate(cats):
        cat_info.append({'name': cat, 'id': i})

    image_set_path = os.path.join(DATA_PATH, 'val.txt')

    ret = {'images': [], 'annotations': [], "categories": cat_info}
    i = 0
    for line in open(image_set_path, 'r'):
        line = line.strip('\n')
        i += 1
        # if your file name contains unique id, just use it instead
        image_id = int(i)
        image_info = {'file_name': '{}'.format(line), 'id': image_id}
        ret['images'].append(image_info)
        anno_path = line.replace('.jpg', '.txt')
        anns = open(anno_path, 'r')
        img = cv2.imread(line)
        height, width = img.shape[0], img.shape[1]
        for ann_id, txt in enumerate(anns):
            tmp = txt[:-1].split(' ')
            cat_id = tmp[0]
            box = [float(tmp[1]), float(tmp[2]), float(tmp[3]), float(tmp[4])]
            bbox = _darknet_box_to_coco_bbox(box, img_width=width, img_height=height)
            area = round(bbox[2] * bbox[3], 2)
            # coco annotation format
            ann = {'image_id': image_id,
                'id': int(len(ret['annotations']) + 1),
                'category_id': int(cat_id),
                'bbox': bbox,
                'iscrowd': 0,
                'area': area}
            ret['annotations'].append(ann)
    out_path = os.path.join(DATA_PATH, 'annotations.json')
    json.dump(ret, open(out_path, 'w'))