search

wang-xinyu / tensorrtx

Implementation of popular deep learning networks with TensorRT network definition API
MIT License
6.89k stars 1.76k forks source link

INT8 calibration for yolov5 lowers accuracy significantly #753

Closed kbegiedza closed 2 years ago

kbegiedza commented 2 years ago

Env

Your problem

I've created my *.engine from previously trained *.pt. For INT8 I've used ~10% of dataset to calibrate model.

Model converted with FP16 precision outputs correct classes with high accuracy, but model with INT8 had very poor accuracy (detections with probability < 0.1).

wang-xinyu commented 2 years ago

AFAIK, tensorrt has bad int8 performance on layers like SiLU.

You can try old version like tensorrt7.0. Or try another model with only RELU as activation.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Lenan22 commented 1 year ago

Please refer to our open source quantization tool ppq, the quantization result is better than the quantization tool that comes with tensorrt, almost the same as the float32 model. https://github.com/openppl-public/ppq/blob/master/md_doc/deploy_trt_by_api.md