search

NVIDIA / TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
https://developer.nvidia.com/tensorrt
Apache License 2.0
10.85k stars 2.14k forks source link

out of memory failure of TensorRT 10.5 when running flux dit on GPU L40S #4214

Open QZH-eng opened 1 month ago

QZH-eng commented 1 month ago

Description

I tried to convert the Flux Dit model on L40S with TensorRT10.5, and found that the peak gpu memory exceeded 46068MiB, but 23597MiB gpu memory was occupied during inference. Is this normal? If normal, what measures can be taken to reduce the gpu memory usage during model conversion so that Flux TensorRT inference can be run normally in the L40S

[10/17/2024-11:07:02] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 22681 MiB, GPU 49917 MiB

Environment

TensorRT Version: 10.5

NVIDIA GPU : L40S

NVIDIA Driver Version: 535.129.03

CUDA Version: 12.2

CUDNN Version:

Operating System:

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

algorithmconquer commented 1 month ago

@QZH-eng I also encountered this OOM (Out of Memory) issue in L40s.

algorithmconquer commented 1 month ago

@QZH-eng I encountered OOM (Out of Memory) issue during inference, Specifically ， when executing "engine_from_bytes(bytes_from_path(self.engine_path))" this step, an OOM (Out of Memory) error occurs.Can you share your code?

QZH-eng commented 1 month ago

@QZH-eng I encountered OOM (Out of Memory) issue during inference, Specifically ， when executing "engine_from_bytes(bytes_from_path(self.engine_path))" this step, an OOM (Out of Memory) error occurs.Can you share your code?

I encountered out of gpu memory when I was converting models on the L40S, when execute trtexec to convert onnx to BF16 plan on the command line.

yuanyao-nv commented 3 weeks ago

You can try quantization with modelopt to reduce the engine size.

asfiyab-nvidia commented 2 weeks ago

@QZH-eng the flux demo should now run on L40S as we have added memory optimizations in release/10.6. Can you please try again and update here?