search

NVIDIA / TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
https://developer.nvidia.com/tensorrt
Apache License 2.0
10.55k stars 2.1k forks source link

Could not find any implementation for node /backbone/layers.0.1.0/attn/attn/MatMul for SegFormer onnx model for int8 precision #4072

Open NannilaJagadees opened 1 month ago

NannilaJagadees commented 1 month ago

Description

I am using this calibration script to generate the calib cache file for Segformer onnx model. But facing this issue while generating calib cache.

image

But I am able to generate engine file from trtexec without the calib file, with this command and didn't face any issues.

trtexec --onnx=segformer.onnx --saveEngine=segformer.engine --int8 --useCudaGraph --dumpLayerInfo --profilingVerbosity=detailed

How can we calibration for this model?

Environment

TensorRT Version: 8.6.2.3

NVIDIA GPU: Orin Nano 8 GB

CUDA Version: 12.2

CUDNN Version: 8.9.4

Operating System: Jetpack 6.0

Python Version: 3.10

PyTorch Version: 2.3.0

lix19937 commented 3 weeks ago

@NannilaJagadees You can upload the full log about when you use calib table to generate engine .

lix19937 commented 3 weeks ago

But I am able to generate engine file from trtexec without the calib file, with this command and didn't face any issues.

It is highly likely that there is a problem with your calib program.

NannilaJagadees commented 3 weeks ago

Hi @lix19937,

Here the full log for segformer while calibrating

segformer_log.txt

lix19937 commented 2 weeks ago

You can try to close --explicit-batch.

lix19937 commented 2 weeks ago

Further more, you can use Polygraphy to calib, https://github.com/NVIDIA/TensorRT/tree/release/10.3/tools/Polygraphy/examples/api/04_int8_calibration_in_tensorrt

NannilaJagadees commented 2 weeks ago

Hi @lix19937

I tried using polygraphy but faced the same issue image