search

NVIDIA / TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
https://developer.nvidia.com/tensorrt
Apache License 2.0
10.8k stars 2.13k forks source link

[Issue still exists in TRT 10.4] Fail to build the engine if input tensor number > 2^31 #4123

Open dzzhang96 opened 2 months ago

dzzhang96 commented 2 months ago

Description

The release note of 10.0.1 mentioned that the issue “UNets with tensors containing >2^31 elements may fail during the engine building step” was fixed. However, my model (nnUNet) has 1x32x512x512x512 input and it still failed to build the engine in TRT 10.4.

Environment

TensorRT Version: 10.4

Relevant Files

Model link: I uploaded a dummy model for testing here.

Similar case https://github.com/NVIDIA/TensorRT/issues/3815 https://github.com/NVIDIA/TensorRT/issues/4004

moraxu commented 1 month ago

@dzzhang96 can you provide your OS and the exact command/snippet you use to build the engine? I'll instance an internal bug at that point, thanks.

dzzhang96 commented 1 month ago

@moraxu Hi, thanks for the reply. I am using Ubuntu 22.04, NVIDIA-SMI: 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0, RTX A4500 with 16G RAM. Sorry I cannot share my codes here but you may use trtexec to reproduce the error. :) Thanks again

moraxu commented 1 month ago

@dzzhang96 I was told that Conv doesn't support > INT32_MAX volume size (see https://docs.nvidia.com/deeplearning/tensorrt/operators/docs/Convolution.html#volume-limits). The "fix" in release note of 10.0.1 may refer to other operators/cases.

dzzhang96 commented 1 month ago

@moraxu Hi thanks for the useful information! I assume that TensorRT is using this Conv operator (nvinfer1::IConvolutionLayer Class) when running inference. Because I did not use it directly in my codes. In the release note, it is hard to be convinced that a unet structure model does not use a Conv operator.

moraxu commented 1 month ago

Yes, correct.

In the release note, it is hard to be convinced that a unet structure model does not use a Conv operator.

Makes sense, we can improve the release note moving forward.

dzzhang96 commented 1 month ago

@moraxu Thanks! It would be very helpful if the input tensor number could be > 2^31, which means we can input the original 512x512x512 image for inference without losing any resolution.

moraxu commented 1 month ago

Noted, although the 2^31 limitation is from an internal library that we depend on..

pieris98 commented 1 month ago

@moraxu This is crippling with any model if you use high resolution inputs. What is the alternative solution? Only to modify the architecture and downsample everything?

moraxu commented 1 month ago

Unfortunately there might be no other solution in this case for now