TensorRT EP could not open shared library from ?

[EternalSaga]

Describe the bug When I was running yolov5 onnx modle on jetson xavier nx with onnxruntime 1.10 version, I encountererd this error.

what():  /home/robin/DataDisk/buildFromSource/onnxruntime/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc:554 onnxruntime::TensorrtExecutionProvider::TensorrtExecutionProvider(const onnxruntime::TensorrtExecutionProviderInfo&) [ONNXRuntimeError] : 11 : EP_FAIL : TensorRT EP could not open shared library from �

Urgency Normal

System information

• OS Platform and Distribution (Linux Ubuntu 18.04):
• ONNX Runtime installed from (source or binary): From Source
• ONNX Runtime version: 1.10
• GCC/Compiler version (if compiling from source): onnxruntime is build by gcc7.5, my application is build by gcc11.1
• CUDA/cuDNN version: 10.2
• GPU model and memory: Jetson Xavier NX with L4t 32.6.1

To Reproduce I checked out the onnxruntime git tag to 1.10. I don't use the latest 1.11 version because 1.11 needs tensor rt 8.2 but the tensor rt version on jetson is 8.0 at present. Apparently, the 1.11 version cannot be built successfully with the --use_tensorrt flag.
I build onnxruntime from source on jetson xavier nx, the build command is

./build.sh --config Release --update --build --parallel \
 --build_shared_lib -use_tensorrt \
 --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu \
 --tensorrt_home /usr/lib/aarch64-linux-gnu

After building, I copied all lib.so and include directory to a single folder. Then I linked these files manually to build my application.

set(ONNXRUNTIME_INCLUDE_DIR "${ONNX_INSTALL_DIR}/include" "${ONNX_INSTALL_DIR}/include/onnxruntime/core/session")

find_library(ONNXRUNTIME_LIBRARY NAMES onnxruntime 
    HINTS ${ONNX_INSTALL_DIR}
    PATH_SUFFIXES lib64 lib
    PATHS ${ONNX_INSTALL_DIR}
    DOC "ONNXRUNTIME - Library")
find_library(ONNXRUNTIME_TENSORRT NAMES onnxruntime_providers_tensorrt 
    HINTS ${ONNX_INSTALL_DIR}
    PATH_SUFFIXES lib64 lib
    PATHS ${ONNX_INSTALL_DIR}
    DOC "ONNXRUNTIME Tensorrt - Library")

target_include_directories(OnnxDetector PRIVATE ${ONNXRUNTIME_INCLUDE_DIR})
target_link_libraries(OnnxDetector PRIVATE ${ONNXRUNTIME_LIBRARY} ${ONNXRUNTIME_TENSORRT})

The application reference codes and models are comes from this repo
I tried to use tensorrt instead of normal cuda inference.

sessionOptions = Ort::SessionOptions();
std::vector<std::string> availableProviders = Ort::GetAvailableProviders();
auto trtAvailable = std::find(availableProviders.begin(), availableProviders.end(), "TensorrtExecutionProvider");
OrtTensorRTProviderOptions tensorrtOpt;
tensorrtOpt.device_id = 0;
if (device == ONNXDevices::TENSOR_RT && (trtAvailable != availableProviders.end())){
        std::cout << "Inference device: Tensor rt" << std::endl;
        sessionOptions.AppendExecutionProvider_TensorRT(tensorrtOpt);
}

After building, the application show the error as above. Furthermore, this application could under the normal cuda successfully. Expected behavior I hope the application can output inference results.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context yolov5

[stevenlix]

I run the model yolov5m.onnx in Nvidia V100 GPU and the inference succeeded. Will check Xavier. It seems like library path issue.

[EternalSaga]

I run the model yolov5m.onnx in Nvidia V100 GPU and the inference succeeded. Will check Xavier. It seems like library path issue.

It's wired that the link path is a '�'. It's hard for me to find the linking path.

[aaost]

FWIW, setting engine decryption in OrtTensorRTProviderOptions to false fixed the issue for me: trtOpts.trt_engine_decryption_enable = false;

[lichuangcheng]

Using OrtTensorRTProviderOptionsV2 instead of OrtTensorRTProviderOptions can solve this problem, as OrtTensorRTProviderOptions is a C structure whose initialization values are mostly undefined. For specific usage, you can refer to the official example: click here for C API example.