calculate mAP after trained subset of COCO with dont_show in .names

[PimwipaV]

Hello Alexey,

I want a yolov3 model that only detect a subset (14 classes) of COCO, so I trained it by adding dont_show to the lines in coco.names that I dont want, as you mentioned a new functionality recently. I trained with classes =80, with the changes in .names, with their original annotation for more than 60 000 iterations. When I stop, the avg.loss was still around 3.4 but it doesn't decrease anymore.

the mAP I got from your build-in map command is only 0.73% and from pycocotools is 0.6%

What did I do wrong?

I am confused now. For those dont_show classes, there are no TP but a lot of FP. I guess the problem must come from the different class names in coco.names when it is used as ground truth for evaluation. but your built-in map only uses the coco.names I provide during training right? So it should be correct. I understood that the training was done by class id, not class names?

It'd be great if you can help me think this through. Could you suggest how do I improve mAP? I was looking at cocoeval.py to see if I can change catIds parameters, or can i adjust something in your build-in map command? or is it something during training?

in the screenshot, all ap is 0.00% but the overall ap comes up to 0.73% because class person is still at its best ap 57% it is one of the 14 classes I want.

[AlexeyAB]

@Pimpwhippa Hi,

so I trained it by adding dont_show to the lines in coco.names that I dont want, as you mentioned a new functionality recently.

You can use dont_show without training, just with default weights-file: https://pjreddie.com/media/files/yolov3.weights

---

I understood that the training was done by class id, not class names?

Yes.

---

I was looking at cocoeval.py to see if I can change catIds parameters, or can i adjust something in your build-in map command?

Can you show how did you use cocoeval / pycocotool in details?

---

I want a yolov3 model that only detect a subset (14 classes) of COCO

Do you use default MS COCO dataset without any changes? That you got by using https://github.com/AlexeyAB/darknet/blob/master/scripts/get_coco_dataset.sh

[PimwipaV]

@AlexeyAB Thank you for your quick response!

You can use dont_show without training, just with default weights-file: https://pjreddie.com/media/files/yolov3.weights

Yes, I figured that out only after I finished training - -' But I also tried measuring mAP of yolov3.weights against my cocodontshow.data, and it gets the same AP 0.73%, which is not unusual I suppose, because it was trained with a different .data, if I haven't missed any crucial point.

Can you show how did you use cocoeval / pycocotool in details?

I just followed this https://github.com/AlexeyAB/darknet/issues/2140

1. Generate results file ./darknet detector test cfg/cocodontshow.data cfg/cocodontshow.cfg cocodontshow_last.weights -thresh 0.005 -dont_show -ext_output < /objectstodetect/cocodataset/val2017/test.txt > result.txt
2. Convert results to json file that could be parse by pycocotools. I use ydixon's script.
3. Run pycocotools, following https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocoEvalDemo.ipynb didn't change anything there

Do you use default MS COCO dataset without any changes? That you got by using https://github.com/AlexeyAB/darknet/blob/master/scripts/get_coco_dataset.sh

I downloaded MS COCO dataset directly from their website, and didn't change anything there either.

Furthermore, after I got ap 0.73% from default weights, I was checking map of yolov3.weights against its coco.data. I got an error, cannot open coco_testdev: where is coco_testdev? Is that for the competition?

Thank you so much!

[AlexeyAB]

@Pimpwhippa

I downloaded MS COCO dataset directly from their website, and didn't change anything there either.

• Did you convert labels to Yolo format?

• Run training with flag -show_imgs do you see correct bounded boxes of objects?

• This script downloads Dataset from mscoco site, and ms-coco labels in yolo format from pjreddie's-site: https://github.com/AlexeyAB/darknet/blob/master/scripts/get_coco_dataset.sh

Furthermore, after I got ap 0.73% from default weights, I was checking map of yolov3.weights against its coco.data. I got an error, cannot open coco_testdev: where is coco_testdev? Is that for the competition?

Yes.

Comment test_dev and un-comment 5k: https://github.com/AlexeyAB/darknet/blob/5ec35922d5215e11466a9bb1602f81d1746ccbe5/cfg/coco.data#L3-L4

[PimwipaV]

Did you convert labels to Yolo format?

yes ofcourse, with this https://bitbucket.org/yymoto/coco-to-yolo/src/master/

Run training with flag -show_imgs do you see correct bounded boxes of objects?

i haven't tried that. I'll do now. I'll let you know later

This script downloads Dataset from mscoco site, and ms-coco labels in yolo format from pjreddie's-site: https://github.com/AlexeyAB/darknet/blob/master/scripts/get_coco_dataset.sh

nope i didn't use that. but i did download from mscoco.org website and converted as mentioned above

Yes. Comment test_dev and un-comment 5k:

ok will do that now too.

thank you so much!! again...you're so great :) will come back with answers soon...

[PimwipaV]

ofcourse i forgot to change cfg to yolov3.cfg!!!

[PimwipaV]

ok i changed to the correct cfg but it's still ap 0.96% with person at 76%

[AlexeyAB]

Did you change conversion script to get labels only for require 14 classes, or do you train with all 80 labels in txt-files?

[PimwipaV]

all 80 labels in txt-files. i mean the annotations/instance_val2017.json

[PimwipaV]

oh and i thought i could change annotations into adding class id 80, 81, 82 and move there like..i dont remember correctly; bicycle, truck, bus; classes I want dont_show. so that id 1,3, 16,17,18 is dont_show bicycle, dont_show truck, dont_show bus so then i wont get so many FPs., and thus improve mAP... would that work?

[AlexeyAB]

If you train by using all 80 classes, then I don't know why do you get low AP for all classes except Person.

---

If you want to detect only some of classes, then you should use:

• either default cfg & wegihts files https://pjreddie.com/media/files/yolov3.weights with dont_show before unnecessary classes in coco.names

• or train your own model with classes=14 in your cfg-file and with renamed classes_id from 1,3,16,17, ... to 0,1,2,...,13 in txt-label-files

[PimwipaV]

ok thank you very much. I'll try both ways. Will let you know when I have any updates.

[PimwipaV]

1583