TensorRT EP failed to set INT8 dynamic range.

[Slyne]

Describe the bug Followed the examples: https://github.com/microsoft/onnxruntime-inference-examples/blob/main/quantization/image_classification/trt/resnet50/e2e_tensorrt_resnet_example.py I add the quantization steps into my pipeline. It can collect the calibration data. However, it fails when running with Tensorrt Execution Provider.

Urgency None

System information Dockerfile: https://github.com/microsoft/onnxruntime/blob/master/dockerfiles/Dockerfile.tensorrt With ARG TRT_VERSION=21.07 changed to ARG TRT_VERSION=21.12

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 18.04
• ONNX Runtime installed from (source or binary):
• ONNX Runtime version:
• Python version:
• Visual Studio version (if applicable):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version:
• GPU model and memory: V100-16G

Screenshots

Additional context Attach the calibration files generated: calibration.zip

BTW, building with the 21.07 version docker will fail.

[stevenlix]

from the error message, it seems min value of tensor 3398 is infinite and I didn't see dynamic range of 3398 in the calibration table. What model do you try to calibrate? Would you mind to share it?

[Slyne]

from the error message, it seems min value of tensor 3398 is infinite and I didn't see dynamic range of 3398 in the calibration table. What model do you try to calibrate? Would you mind to share it?

Sorry, I removed the tensors with inf, -inf when writing the calibration table after I find this issue. The original calibration files: calibration.zip

The codes are from: https://github.com/wenet-e2e/wenet/tree/main/wenet The script is edited from: https://github.com/wenet-e2e/wenet/blob/main/wenet/bin/recognize_onnx.py recognize_onnx_calibrate.zip

[stale[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[piedras77]

I am facing the same issue. Were you able to find a solution?