Open luoshiyong opened 4 months ago
devicec : nvidia NX
1.using trt --fp16
/usr/src/tensorrt/bin/trtexec --onnx=best.onnx --workspace=4096 --saveEngine=best.engine --fp16
the result of infer speed is 36.8ms
--int8
means Enable int8 precision, in addition to fp32.
--fp16
means Enable fp16 precision, in addition to fp32. Maybe your bad q-dq
setting. You can compare layer profile detail among two build logs.
--int8
means Enable int8 precision, in addition to fp32.--fp16
means Enable fp16 precision, in addition to fp32. Maybe your badq-dq
setting. You can compare layer profile detail among two build logs.
i have try varies command such as "--bset" or "--int8 --fp16 --noTF32" ,but no use, i fully understand the option "--int8 --int16 --best". What's mean the bad q-dq setting ? what i get is some layers is not applicable to int8 quant.
If you use trt ptq quant(implicitly quantized), you can ignore q-dq setting.
but i use the tensorrt explicitly quant, insert q/dq node, i want to know how to insert proper q/dq for faster perfomance, can you give me some advice?
@luoshiyong Ref ptq fusion result.
devicec : nvidia NX 1.using trt --fp16
/usr/src/tensorrt/bin/trtexec --onnx=best.onnx --workspace=4096 --saveEngine=best.engine --fp16
the result of infer speed is 36.8ms/usr/src/tensorrt/bin/trtexec --onnx=best.onnx --saveEngine=v8s_ptq.engine --int8 --workspace=4096
the result of infer speed is : 39.5ms