search

Tianxiaomo / pytorch-YOLOv4

PyTorch ,ONNX and TensorRT implementation of YOLOv4
Apache License 2.0
4.47k stars 1.49k forks source link

Did you check accuracy of YOLOv4 by using your implementation on MSCOCO? #190

Open AlexeyAB opened 4 years ago

AlexeyAB commented 4 years ago

@Tianxiaomo Hi,

test

li10141110 commented 4 years ago

+1

Jonathanjopk commented 4 years ago

+1

alrzmshy commented 4 years ago

This is what I got with the yolov4.pth, 608x608 input: predictions_onnx

ersheng-ai commented 4 years ago

This is result on COCO validation set (val2017), input is 416*416. It seems it does not reach optimal accuracy of the original DarkNet model

 conf_thresh = 0.001   NMS_IOU_thresh = 0.6

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.42601
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.65587
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.45796
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.21062
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.48669
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.60758
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.32180
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.51224
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.55351
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.32155
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.62874
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.74219

This is result from https://github.com/AlexeyAB/darknet/issues/5354

###############################################################################
#       YOLOV4 416x416 CODALAB res COCO2017 VAL               #
###############################################################################

overall performance
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.471
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.710
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.510
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.278
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.525
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.636
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.357
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.561
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.587
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.382
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.642
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.772
AlexeyAB commented 4 years ago

@ersheng-ai

ersheng-ai commented 4 years ago

@ersheng-ai

  • Did you use converted or trained weights on Pytorch?
  • Do you use keeping aspect ration of image during resizing? YOLOv4 doesn't keep aspect ration by default.

Pretrained DarkNet weights were used to generate pth. I pad zeros to images to keep aspect ratios.

AlexeyAB commented 4 years ago

@ersheng-ai Try to use resizing without keeping-aspect-ratio/padding-zeros, will accuracy be better?

ersheng-ai commented 4 years ago

@ersheng-ai Try to use resizing without keeping-aspect-ratio/padding-zeros, will accuracy be better?

I will try it

ersheng-ai commented 4 years ago

Zero padding is removed so that images of any ratio are squashed or stretched to 416 * 416 Now the results look much closer to https://github.com/AlexeyAB/darknet/issues/5354

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.46605
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.70384
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.50477
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.26692
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.52448
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.62891
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.34238
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.54948
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.59087
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.38072
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.65378
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.76268
AlexeyAB commented 4 years ago

@ersheng-ai

So there is no big drop in accuracy. Great!

Do you use padding = SAME or VALID for conv and maxpooling(in SPP-block) in YOLOv4? Darknet always uses SAME padding for conv/maxpool.

ersheng-ai commented 4 years ago

@ersheng-ai

  • github.com/AlexeyAB/darknet - 47.1% AP - 71.0% AP50
  • Tianxiaomo/pytorch-YOLOv4 - 46.6% AP (-0.5) - 70.3% AP50 (-0.7)

So there is no big drop in accuracy. Great!

Do you use padding = SAME or VALID for conv and maxpooling(in SPP-block) in YOLOv4? Darknet always uses SAME padding for conv/maxpool.

I think the Pytorch model strictly follows the routines. The slight drop of accuracy may due to post-process like NMS. We are looking into it.

hujunchao commented 4 years ago

Hi,@ersheng-ai. Thank you for your great job! And I have a question! This project seems to only use yolov4' network and mosaic, but donot use Eliminate grid sensitivity，IoU threshold, CIOU loss and so on. This project get 46.6% AP while darknet yolov4 only get 47.1% AP in coco val-2017. So what matters most in getting AP as high as darknet ?

guyuchao commented 4 years ago

@ersheng-ai

  • github.com/AlexeyAB/darknet - 47.1% AP - 71.0% AP50
  • Tianxiaomo/pytorch-YOLOv4 - 46.6% AP (-0.5) - 70.3% AP50 (-0.7)

So there is no big drop in accuracy. Great! Do you use padding = SAME or VALID for conv and maxpooling(in SPP-block) in YOLOv4? Darknet always uses SAME padding for conv/maxpool.

I think the Pytorch model strictly follows the routines. The slight drop of accuracy may due to post-process like NMS. We are looking into it.

How the performance obtained, train from cspdarknet imagenet pretrained model or yolov4 coco pretrained model?

DaChaoXc commented 3 years ago

why pyorch visin should use conf_thresh = 0.001 to compute mAP?

jicongh commented 3 years ago

Hi,@ersheng-ai. Thank you for your great job! I have one question: your computer hardware information？ how long time you train a complete coco project?

jicongh commented 3 years ago

our great job! I have one question: your computer hardware information？ how long time you train a complete coco project?

Is there any code to reproduce this mAP?

abhiagwl4262 commented 3 years ago

@ersheng-ai How do I run validation on coco ?

GuoQuanhao commented 2 years ago

How to evaluate, can you provide the command?

kazu-gra commented 1 year ago

What files did you use and what commands did you use to implement your evaluation? Any other changes from the original gituhub code would be appreciated.