Convert a quantized conv3d model failed

[YohannXu]

Description

I'm using pytorch quantization toolkit to quantize my model, which has some conv3d module. The QAT procedure is OK. But when i use trtexec to convert the onnx with Q/DQ pairs to engine, it failed! I have written some simple code to reproduce the error.

Environment

TensorRT Version: 8.2.5.1 NVIDIA GPU: TITAN RTX NVIDIA Driver Version: 470.103.01 CUDA Version: 11.4 CUDNN Version: Operating System: ubuntu 20.04 Python Version (if applicable): 3.8.10 Tensorflow Version (if applicable): PyTorch Version (if applicable): 1.9.1+cu102 Baremetal or Container (if so, version):

Relevant Files

import torch
import torch.nn as nn
from pytorch_quantization import quant_modules
import pytorch_quantization.nn as quant_nn
quant_modules.initialize()

model = nn.Conv3d(3, 16, kernel_size=(3, 3, 3), stride=(1, 1,1), padding=(1, 1, 1))
quant_nn.TensorQuantizer.use_fb_fake_quant = True
torch.onnx.export(model, torch.randn(1, 3, 20, 20, 20), 'conv3d_quant.onnx', opset_version=13)

Steps To Reproduce

use above code to generate a conv3d onnx with q/dq pair, it should look like this.

Then use

trtexec --onnx=conv3d_quant.onnx --int8 --noTF32 --saveEngine=conv3d_quant.engine

it will throw a error

I guess tensorrt doesn't support conv3d onnx with q/dq pair？

[zerollzeng]

I can't export the onnx using your script, can you share the onnx model here?

$ python3 2125.py
Traceback (most recent call last):
  File "2125.py", line 8, in <module>
    torch.onnx.export(model. torch.randn(1, 3, 20, 20, 20), 'conv3d_quant.onnx', opset_version=13)
  File "/home/scratch.zeroz_sw/miniconda3/envs/transformer/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'QuantConv3d' object has no attribute 'torch'

[zerollzeng]

cc @ttyio

[YohannXu]

@zerollzeng i fixed code above, you can try it again

[nvpohanh]

could you try adding Q/DQ after the Conv? I am not sure if we have tactics for INT8-input FP32-output 3D convs (will need to check internally).

[YohannXu]

you mean a mode like this? it doesn't work. @nvpohanh

[nvpohanh]

Thanks for trying. We will debug this. @zerollzeng Could you give it a try with TRT 8.2 and TRT 8.4? Let's see if it's already fixed in TRT 8.4. Thanks

[zerollzeng]

@YohannXu I still met error using your export code:

import torch
from pytorch_quantization import quant_modules
import pytorch_quantization.nn as quant_nn
quant_modules.initialize()

model = nn.Conv3d(3, 16, kernel_size=(3, 3, 3), stride=(1, 1,1), padding=(1, 1, 1))
quant_nn.TensorQuantizer.use_fb_fake_quant = True
torch.onnx.export(model, torch.randn(1, 3, 20, 20, 20), 'conv3d_quant.onnx', opset_version=13)

$ python3 2125.py
Traceback (most recent call last):
  File "2125.py", line 6, in <module>
    model = nn.Conv3d(3, 16, kernel_size=(3, 3, 3), stride=(1, 1,1), padding=(1, 1, 1))
NameError: name 'nn' is not defined

[zerollzeng]

Looks like they are not modified.

[YohannXu]

sry, i forgot to import torch.nn, it now works @zerollzeng

[zerollzeng]

I can't reproduce it in TRT 8.4, can you try 8.4?

TensorRT-8.4.1.5/bin/trtexec --onnx=conv3d_quant.onnx --int8

[YohannXu]

i tried 8.4, but the error still occur. @zerollzeng maybe you can try my onnx model model.zip

[YohannXu]

I update the library, it now works. Thanks @zerollzeng

[YohannXu]

For a simple onnx above, it now works. But for a complex onnx, the error occur again at the first conv3d op. model.zip @zerollzeng