(Be sure to let me know if you found the problem.)

[adrianosantospb]

Hi, guys.

So... I tested the YOLO V3 (SPP and Tiny) conversion and I noticed that the model accuracy is lower after conversion. Currently, this model is 99.9% accurate, and it has only one class. And it's not only missing detection but "ghost detections" as well (strangers object detections).

[jkjung-avt]

@adrianosantospb

Thanks for bringing this up. I do think mAP of my TensorRT yolov3-tiny implementations are very low. Do let me know if you have any thoughts or suggestions.

TensorRT engine	mAP @ IoU=0.5:0.95	mAP @ IoU=0.5	FPS on Nano
yolov3-tiny-288 (FP16)	0.077	0.158	35.8
yolov3-tiny-416 (FP16)	0.096	0.202	25.5

[ROBYER1]

@adrianosantospb

Thanks for bringing this up. I do think mAP of my TensorRT yolov3-tiny implementations are very low. Do let me know if you have any thoughts or suggestions.

TensorRT engine mAP @ IoU=0.5:0.95 mAP @ IoU=0.5 FPS on Nano yolov3-tiny-288 (FP16) 0.077 0.158 35.8 yolov3-tiny-416 (FP16) 0.096 0.202 25.5

Was there any fix for the low mAP results in v3 tiny?

[philipp-schmidt]

@adrianosantospb Can you share more info on the "ghost detections" that you mention? I'm having similar problems with the TensorRT implementation in my repo (https://github.com/isarsoft/yolov4-triton-tensorrt), which is very similar to the @jkjung-avt implementation. The error is VERY difficult to track down so I'm very happy someone else experiences them as well and we have more examples to work with.

Regarding the degraded precision after conversion: to some extent this is expected due to FP16 conversion, but the difference is to big currently in both implementations. Im wondering if the yolo layer plugin is the cause, but I have a different TensorRT repo with a different network which has similar problems (no plugins).

I would like to track down this issue so any info that you can share is very much appreciated. How do you test mAP? How many GPUs do you use? When do the ghost detections occur? Do you run the engine natively or in a framework/deployment?

[adrianosantospb]

Hi, @philipp-schmidt . Sure I can. We can work together to solve it. :) So... this problem happens when the detection model has low accuracy. I still don't know why, but when we convert the YOLO to TensorRT using the python' approach, we get worse or not enough result. In my case, for example, I have a YOLO model trained with exceptional accuracy. But when I converted to TensorRT, the result was very bad... I'm been testing with the C++ approach and the detection was quite good (of course it isn't as the full model, but the result is enough). Take a look at this repo (https://github.com/ceccocats/tkDNN). But I'm been noticed that using this repo, I've got a memory leak. So... that is it. I'm still working to solve these problems.

[philipp-schmidt]

Would you mind trying another implementation and tell us whether the accuracy is low as well? https://github.com/isarsoft/yolov4-triton-tensorrt

It should be relatively straigthforward to build the engine if you have the darknet weights. Ping me if you need help with converting the weights, I can help. You don't necessarily need to deploy to Triton.

Instructions for converting weights

[ROBYER1]

Has anyone here used an ONNX converter for Yolo v3 tiny that doesn't reduce the accuracy? And also can we confirm that the ONNX file is fine before then converting to Tensorrt to eliminate whether it is the ONNX conversion or the Tensorrt conversion?

I am currently using the ONNX files with Unity Barracuda but am having difficulties reading Yolo v3 tiny outputs to evaluate the converted model however I am noticing inconsistent results with detected classes already.

[jkjung-avt]

I'm closing this issue and will continue to track the "trt yolo mAP dropping" problem here: https://github.com/jkjung-avt/tensorrt_demos/issues/315