C++ inference with GPU (CUDA)

[wvalcke]

Describe the issue

Small program

int main(int argc, char *argv[]) { Ort::Env env; Ort::SessionOptions session_options; OrtCUDAProviderOptions cuda_options; cuda_options.device_id = 0; session_options.AppendExecutionProvider_CUDA(cuda_options); Ort::Session session = Ort::Session(env, L"rim.onnx", session_options); std::cout << "End reached" << std::endl; }

running this program, we have no output (also not any error or something like that). When taking out the AppendExecutionProvider_CUDA call it works fine. Any idea how to debug this kind of problem? I'm using VS 2019 & onnxruntime 1.13.1 with cuda 11.4

To reproduce

int main(int argc, char *argv[]) { Ort::Env env; Ort::SessionOptions session_options; OrtCUDAProviderOptions cuda_options; cuda_options.device_id = 0; session_options.AppendExecutionProvider_CUDA(cuda_options); Ort::Session session = Ort::Session(env, L"rim.onnx", session_options); std::cout << "End reached" << std::endl; }

Urgency

No response

Platform

Windows

OS Version

10

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.13.1

ONNX Runtime API

C++

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

No response

[pranavsharma]

Did you install the requirements here https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements?

[wvalcke]

Hi,

thanks for the reply. Some more information: the software is compiled from git source (release 1.13.1) During compilation i also let it build the python wheel. After the compilation the python wheel was installed, and runs fine both for CPU and GPU. This proves that the build is fine. With the simple C++ test the exe starts, but just stops after some seconds, dumps a crashdump, nothing more. No output, nopthing. Doing some C++ samples using inference on CPU works fine. When using GPU, i think something is loaded dynamically, but goes wrong at some point. But how to find out? I did the same tests also under Linux, both GPU & CPU with the same c++ example, no problem at all.

[wvalcke]

The error is actually produced on the following code, (checked via the dmp file)

inline SessionOptions& SessionOptions::AppendExecutionProvider_CUDA(const OrtCUDAProviderOptions& provider_options) { ThrowOnError(GetApi().SessionOptionsAppendExecutionProviderCUDA(p, &provider_options)); return *this; }

[wvalcke]

I finally got it working. I needed to copy the following dll's in the same folder as the exe

• onnxruntime.dll
• onnxruntime_providers_cuda.dll
• onnxruntime_providers_shared.dll Then it started working. Although the folders with the DLL's were in the PATH, it gave issues. Strange why this is not happening with the python version.

[pranavsharma]

Thanks. This might be a good opportunity for us to improve the experience for developers who build from the source by improving the documentation. cc @natke

[natke]

@pranavsharma Is the above mentioned flow (ie copying the dlls into the same folder as the exe) the way we intend developers to build from source? Could we modify the build script to implement this if so?

[zhanggd001]

I finally got it working. I needed to copy the following dll's in the same folder as the exe

• onnxruntime.dll
• onnxruntime_providers_cuda.dll
• onnxruntime_providers_shared.dll Then it started working. Although the folders with the DLL's were in the PATH, it gave issues. Strange why this is not happening with the python version.

I have the same issue (Win10, VS2017 (C++), onnxruntime 1.12.1). When running the program without onnxruntime_providers_cuda.dll and onnxruntime_providers_shared.dll, the network inference is based on CPU. After copying the following dll's in the same folder as the exe, the inference is based on GPU.

Besides, does onnxruntime provide API for detecting the version and compatibility of CUDA/cuDNN?

[pranavsharma]

Besides, does onnxruntime provide API for detecting the version and compatibility of CUDA/cuDNN?

No.

[lucasjinreal]

I copied dll but still get on output on windows cuda.

sessionOptions.AppendExecutionProvider_CUDA(cuda_options);
    // SessionOptionsAppendExecutionProvider_CUDA(sessionOptions, &cuda_options);
    // sessionOptions.SetGraphOptimizationLevel(ORT_ENABLE_ALL);

    LOG(INFO) << "Running in on GPU!";

the log line didn't print, stoped at add options.

[David-19940718]

The same issue raised on Ubuntu, who can help me?

With the simple C++ test the exe starts, but just stops after some seconds, dumps a crashdump, nothing more. No output, nopthing. Doing some C++ samples using inference on CPU works fine.

For deteil message:

 [W:onnxruntime:, session_state.cc:1169 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
 [W:onnxruntime:, session_state.cc:1171 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.

[prsnkmr]

Hi ,

I created the cudaoptions as explained above by #wvalcke , when i run the make -j8 it does not give any error but when i am trying to run the error its giving the error as mentioned below,Any help or thoughts can be appreciated.

terminate called after throwing an instance of 'Ort::Exception' what(): /home/T7751P1/Downloads/Face_Alingment_CPP/ONNX/onnxruntime/onnxruntime/core/session/provider_bridge_ort.cc:1195 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libonnxruntime_providers_cuda.so: cannot open shared object file: No such file or directory

Thanks

[wxd-0107]

I finally got it working. I needed to copy the following dll's in the same folder as the exe

• onnxruntime.dll
• onnxruntime_providers_cuda.dll
• onnxruntime_providers_shared.dll Then it started working. Although the folders with the DLL's were in the PATH, it gave issues. Strange why this is not happening with the python version.

It works for me, thanks a lot.