The results of COCO evaluation and Darkent evaluation are quite different !

[KKDDD]

I have successfully trained "yolov4-tiny" on the vehicle data set in "bdd 100k" (training: 69133 vehicle images, verification: 9902 vehicle images). In order to compare and detect I use NanoDet-m to compare with it. The two iterated the same number of times: about 300 epochs，all use ImageNet pre-trained weights Nanodet : batchSize=160, iterations=130k yolov4-tiny: batchSize=160, iterations=130k The results of both are as follows:(yolov4 used “-points 101” parameter)

	NanoDet-m	Yolov4-tiny-320
Input	320x320	320x320
map@0.50	38	16.1
map@0.50:0.95	25	10.5

（The evaluation of NanoDet uses the default coco mode, and the evaluation of yolov4-tiny uses method in this website。） The gap between the two is very large as shown in the figure above, but I checked some test results on the verification set, and I found that the effects of the two are similar. whenthe NanoDet homepage comparing with yolov4-tiny on the coco data set, the gap between the two is very small, as shown below.

	NanoDet-m	Yolov4-tiny
Input	320x320	416x416
map@0.50	20.6	21.7

So I consider whether the evaluation method of yolov4-tiny is somewhat different from that of coco, or if I change its input to 320, it will affect the training of the model.

At the same time, I tried to use ./darknet detector valid cfg/coco.data cfg/yolov4.cfg yolov4.weights to generate the detection result in coco format, but I found that all image_id is 0, so I can’t use the coco tool to evaluate it.

As shown in the figure below, some of the detection effects of the two are shown. The upper part is yolo, and the lower part is nanoDet.

It can be seen that the detection gap between the two is not very large

[KKDDD]

I found that darknet supports generating json results in bdd format, so I set eval=bdd in the“.data file(I found this option in the detector.c file), and the final generated json file does indeed contain the image name of BDD, category, score, bbox, but there are some problems, I only use the“car, truck, bus ”three categories in the bdd data set, but in the final generated json file, there is no ”truck“ category, but an extra“ bike ”Category, this makes me wonder, isn’t the final predicted category mapped according to the .names folder?

[KKDDD]

I found that darknet supports generating json results in bdd format, so I set eval=bdd in the“.data file (I found this option in the detector.c file), and the final generated json file does indeed contain the image name of bdd , Category, score, box, but there are some problems, I only use the“car, truck, bus ”three categories in the bdd data set, but in the final generated json file, there is no ”truck“ category, but an extra“ bike ”Category, this makes me wonder, isn’t the final predicted category mapped according to the .names folder?

I found the corresponding category of BDDin line 601 of detector.c. I modified it and generated the correct category. However, after I converted the json in BDDformat into json in the COCOcorresponding format, I tested it through the COCO API, and I found the accuracy Especially low, only 0.5%, I am checking whether it is an error when converting JSON.....

[KKDDD]

Okay, I modified the previous JSON conversion bug, now he successfully converted to COCO format, and completed the accuracy test through COCO API as shown in the table below。 This is all evaluated using COCO API

	NanoDet-m	Yolov4-tiny-320	Yolov4-tiny-416
Input	320x320	320x320	416x416
map@0.50(coco)	38.4	48.4	53.8
map@0.50:0.95(coco)	25.1	30.3	34.3

But when I use darknet's built-in accuracy evaluation commands, such as ./darknet detector valid cfg/coco.data cfg/yolov4.cfg yolov4.weights, the accuracy of yolov4 will become very low. I have checked other similar issues. For the COCO data set, the difference between darknet's evaluation accuracy and COCO's evaluation accuracy is very small. Why is there such a big gap between them? Do you know why? @AlexeyAB

	NanoDet-m	Yolov4-tiny-320	Yolov4-tiny-416
Input	320x320	320x320	416x416
map@0.50	38.4(coco)	16.1(darknet)	18.1(darknet)