jackcaron
commented
1 month ago I have the same error when trying to build a TRT file from an ONNX file. On the same OS version, driver version 560.35.03 and using TensorRT 10.4.
Also, when I'm trying to load a TRT file that was built last week, before upgrading to GCC 13, and I'm getting this stack trace:
#0 0x00007ffff7ea516e in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1 0x00007ffff7ea5d5a in _Unwind_Find_FDE () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#2 0x00007ffff7ea160a in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#3 0x00007ffff7ea307d in _Unwind_RaiseException () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#4 0x00007ffff7cb705b in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007fffbec3c0e5 in ?? () from /usr/lib/x86_64-linux-gnu//libnvinfer.so.10
#6 0x00007fffbecf24c6 in ?? () from /usr/lib/x86_64-linux-gnu//libnvinfer.so.10
#7 0x00007fffbecf41ac in ?? () from /usr/lib/x86_64-linux-gnu//libnvinfer.so.10
#8 0x00007fffbecf53ca in ?? () from /usr/lib/x86_64-linux-gnu//libnvinfer.so.10
#9 0x00007fffd32d6e7a in std::default_delete<nvinfer1::IRuntime>::operator() (this=0x7fffad7fdc58, __ptr=0x7fff740040b0) at /usr/include/c++/13/bits/unique_ptr.h:99
#10 0x00007fffd32d5cae in std::unique_ptr<nvinfer1::IRuntime, std::default_delete<nvinfer1::IRuntime> >::~unique_ptr (this=0x7fffad7fdc58, __in_chrg=<optimized out>)
at /usr/include/c++/13/bits/unique_ptr.h:404
#11 0x00007fffd32d2919 in loadCudaEngine (trtPath="my_file.trt", logger=warning: RTTI symbol not found for class 'TensorRTLogger'
...)
at loading_trt.cpp:170This happens when deallocating a IRuntime after deserializeCudaEngine was called, otherwise it can be deleted (don't know if anything changes in IRuntime, but before upgrading to gcc-13, this was working fine, I also tried keeping the IRuntime alive a bit longer, but it just crashes further in the process).
When using memcheck, matching with this stack trace above, there's this message:
Use of uninitialised value of size 8Edit 1
I was able to fix my bug above my keeping the IRunTime alive longer than the engine. Then I had a secondary logic bug (which is why the first time I tried that it didn't work). But this wasn't needed before, or it just kept going.
As for creating the TRT file from and an .onnx file, the crash above for me only happens if config->setBuilderOptimizationLevel(5); is called. Ignored, or any optimization level below 5 prevents it from crashing.
Hope this helps diagnosing the problem.
yuanyao-nv
zeroepoch
jokla
Description
Tensorrt seg fault when parsing an ONNX model ( yolov8 QAT) with gcc13 installed in Ubuntu 22.04.
Environment
TensorRT Version: 10.3 or olders
NVIDIA GPU: NVIDIA RTX A6000
NVIDIA Driver Version: 560.28.03
CUDA Version: 12.6
CUDNN Version: 8.9.6.50-1+cuda12.2
Operating System:
Container : ubuntu-22.04.Dockerfile + gcc 13 installed
Same issue with nvcr.io/nvidia/tensorrt:24.08-py3 with gcc13 installed on top of it.
Relevant Files
Steps To Reproduce
Commands or scripts:
It seems that the issue is coming from libnvinfer.so.10 and gcc13. The TRT open source version uses a prebuilt nvinfer (from https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/10.3.0/tars/TensorRT-10.3.0.26.Linux.x86_64-gnu.cuda-12.5.tar.gz) , possibly compiled with an older gcc (gcc 8 looking at this table ). The conversion is working on an Orin with Jetpack 6 ( probably because TRT is build with a newer gcc version).
How can I make TRT (and libnvinfer) compatible with gcc13? Also, is there a specific reason why it's only built with an old version of gcc?
Many thanks!
Have you tried the latest release?: Yes, same issue
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (
polygraphy run <model.onnx> --onnxrt): Yes