search

NVIDIA / TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
https://developer.nvidia.com/tensorrt
Apache License 2.0
10.15k stars 2.08k forks source link

Facing "AttributeError: 'tensorrt.tensorrt.INetworkDefinition' object has no attribute 'has_explicit_precision'" when calibrating int8 engine with Polygraphy #3928

Closed Justin020718 closed 3 weeks ago

Justin020718 commented 3 weeks ago

Description

Trying to convert an ONNX model to int8 engine with Polygraphy 0.47.1, saying:

>> polygraphy convert IntraNoAR_decoder_sur_sim.onnx --int8 --data-loader-script ./calib_i.py --trt-min-shapes y_hat:[1,256,23,30] --trt-max-shapes y_hat:[1,256,100,160] --trt-opt-shapes y_hat:[1,256,100,160] --calibration-cache identity_calib.cache -o I_E_int8.engine --verbose
[W] 'colored' module is not installed, will not use colors when logging. To enable colors, please install the 'colored' module: python3 -m pip install colored
[V] Loaded Module: polygraphy | Version: 0.47.1 | Path: ['E:\\anaconda3\\envs\\python3.8\\Lib\\site-packages\\polygraphy-0.47.1-py3.8.egg\\polygraphy']
[V] Loaded Module: tensorrt | Version: 10.0.1 | Path: ['E:\\anaconda3\\envs\\python3.8\\Lib\\site-packages\\tensorrt']
[V] Created calibrator [cache=identity_calib.cache]
[V] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 7685, GPU 1413 (MiB)
[V] [MemUsageChange] Init builder kernel library: CPU +2688, GPU +310, now: CPU 10665, GPU 1723 (MiB)
[V] ----------------------------------------------------------------
[V] Input filename:   E:\DCVC-main\DCVC-DC-encoder-decoder\IntraNoAR_decoder_sur_sim.onnx
[V] ONNX IR version:  0.0.8
[V] Opset version:    17
[V] Producer name:    pytorch
[V] Producer version: 2.3.0
[V] Domain:
[V] Model version:    0
[V] Doc string:
[V] ----------------------------------------------------------------
[V]     Setting TensorRT Optimization Profiles
[V]     Input tensor: y_hat (dtype=DataType.FLOAT, shape=(1, 256, -1, -1)) | Setting input tensor shapes to: (min=[1, 256, 23, 30], opt=[1, 256, 100, 160], max=[1, 256, 100, 160])
[V]     Input tensor: q (dtype=DataType.FLOAT, shape=(1,)) | Setting input tensor shapes to: (min=[1], opt=[1], max=[1])[V]     Input tensor: dummy (dtype=DataType.FLOAT, shape=(1,)) | Setting input tensor shapes to: (min=[1], opt=[1], max=[1])
[I]     Configuring with profiles: [Profile().add('y_hat', min=[1, 256, 23, 30], opt=[1, 256, 100, 160], max=[1, 256, 100, 160]).add('q', min=[1], opt=[1], max=[1]).add('dummy', min=[1], opt=[1], max=[1])]
Traceback (most recent call last):
  File "\\?\E:\anaconda3\envs\python3.8\Scripts\polygraphy-script.py", line 33, in <module>
    sys.exit(load_entry_point('polygraphy==0.47.1', 'console_scripts', 'polygraphy')())
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\tools\_main.py", line 70, in main
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\tools\base\tool.py", line 171, in run
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\tools\convert\convert.py", line 98, in run_impl
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\tools\args\backend\trt\loader.py", line 575, in load_engine_bytes
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\backend\base\loader.py", line 40, in __call__
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\util\util.py", line 694, in wrapped
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\backend\trt\loader.py", line 512, in call_impl
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\util\util.py", line 663, in invoke_if_callable
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\backend\base\loader.py", line 40, in __call__
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\util\util.py", line 694, in wrapped
  File "E:\anaconda3\envs\python3.8\lib\site-packages\polygraphy-0.47.1-py3.8.egg\polygraphy\backend\trt\config.py", line 304, in call_impl
AttributeError: 'tensorrt.tensorrt.INetworkDefinition' object has no attribute 'has_explicit_precision'

Here is my code to generate data:

def load_data():
    for _ in range(500):
        cls = random.choice(class_list)
        path = sample_path + '/' + cls + '/' + random.choice(video_list[cls])
        width, height = get_shape_from_class(cls)
        x, q, dummy = read_one_frame(path, width, height)
        yield {
            "y_hat": np.array(x, dtype=np.float32),
            "q": np.array(q, dtype=np.float32),
            "dummy": np.array(dummy, dtype=np.float32)
        }  # Still totally real data

I've also made another issue, where I was trying to convert with python script. Thank u for ur time. https://github.com/NVIDIA/TensorRT/issues/3926#issue-2338398048

Environment

TensorRT Version: 10.0.1.6 NVIDIA GPU: RTX 3070 laptop NVIDIA Driver Version:

CUDA Version: 12.4

Operating System: Windows 10 Python Version: 3.8 PyTorch Version: 2.3.0

Relevant Files

Model link:

https://drive.google.com/file/d/1O-ZT50QBbaED0ouYgUQGXprYadiPr3P2/view?usp=sharing

Steps To Reproduce

  1. Download "IntraNoAR_decoder_sur_sim.onnx" and make a new script.
  2. Copy and paste following codes: 
    def load_data():
for _ in range(500):
    x = torch.randn(1,256,23,30)
    q = torch.Tensor([random.random()*63]).reshape(1,)
    dummy = torch.Tensor([0]).reshape(1,)
    yield {
        "y_hat": np.array(x, dtype=np.float32),
        "q": np.array(q, dtype=np.float32),
        "dummy": np.array(dummy, dtype=np.float32)
    }

  3. Run following commands:

    polygraphy convert IntraNoAR_decoder_sur_sim.onnx --int8 --data-loader-script ./calib_i.py --trt-min-shapes y_hat:[1,256,23,30] --trt-max-shapes y_hat:[1,256,100,160] --trt-opt-shapes y_hat:[1,256,100,160] --calibration-cache identity_calib.cache -o I_E_int8.engine --verbose

Commands or scripts:

polygraphy convert IntraNoAR_decoder_sur_sim.onnx --int8 --data-loader-script ./calib_i.py --trt-min-shapes y_hat:[1,256,23,30] --trt-max-shapes y_hat:[1,256,100,160] --trt-opt-shapes y_hat:[1,256,100,160] --calibration-cache identity_calib.cache -o I_E_int8.engine --verbose

Have you tried the latest release?: Yes. Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt): Yes.

lix19937 commented 3 weeks ago

@Justin020718

before migration after
INetworkDefinition.has_explicit_precision Explicit precision support is removed in 10.0

Your tensorrt is Version: 10.0.1

ref https://docs.nvidia.com/deeplearning/tensorrt/migration-guide/index.html

Justin020718 commented 3 weeks ago

@lix19937 Thank you for your reply, this is a version issue. After updating to polygraphy 0.49, it can convert successfully without such error.

Justin020718 commented 3 weeks ago

Update to polygraphy 0.49 and it wont happen.