custom model to trt-int8 problem

[tingggggg]

Hi, I converted own yolo model (onnx) to mymodel.trt(FP16) & mymodel-int8.trt(INT8) then use trt_yolo.py to observe the detection results. With same image, FP16 model could detect correctly objects but not INT8 model(could not detect anything).

My model is a mobilenet arch yolo (two ouputs). I calibrated model with about 2,700 images.

Did I missing any detail to lead this error result?

Thank you~

[jkjung-avt]

The quality (coverage) of the calibration set is important. Make sure the 2,700 images cover "the distribution of images you intend to do inference with". For example, you should have both "dark scene" and "bright scene" images in the calibration set, especially if the INT8 model would encounter both kinds of images at inference time.

[tingggggg]

@jkjung-avt Thanks for ur answer. I already check the quality of calibration set (cover all scenarios) but still got same problem. Are there any other steps that need attention?

By the way my backbone of yolo model is depthwise conv, is it the reason lead to this problem?

[jkjung-avt]

I have no idea about why. I also observed big mAP discrepency between FP16 and INT8 TensorRT engines for the "yolov4-608" model.

You could probably try to use polygraphy to compare inference results: ONNX vs. TensorRT FP16 vs. TensorRT INT8. Reference: https://github.com/jkjung-avt/tensorrt_demos/issues/315#issuecomment-817143067

[jkjung-avt]

@tingggggg Does your mobilenet-backboned yolo model require somewhat different preprocessing than what's implemented in this repo? Maybe it's a good idea for you to double check the preprocessing function as well?

Please refer to: https://github.com/jkjung-avt/tensorrt_demos/issues/565#issuecomment-1157632363