OnnxRuntime GPU does not work in WinUI project

[mattleibow]

Describe the bug

I'm trying to make OnnxRuntime GPU library work with .NET MAUI.

I call the method SessionOptions.MakeSessionOptionWithCudaProvider() and got the OnnxRuntimeException: " LoadLibrary failed with error 126 when trying to load onnxruntime_providers_cuda.dll". I checked that dll exists in the output directory.

I can successfully use the library in the following cases:

• MAUI + OnnxRuntime CPU
• WinForm + OnnxRuntime GPU
• Console + OnnxRuntime GPU
• WPF + OnnxRuntime GPU

For that reason, I think that this is a WinUI bug.

Steps to reproduce the bug

1. Create a WinUI Project
2. Add this package: https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.Gpu/
3. Call SessionOptions.MakeSessionOptionWithCudaProvider()

Expected behavior

No response

Screenshots

No response

NuGet package version

None

Packaging type

No response

Windows version

No response

IDE

No response

Additional context

Microsoft.ML.OnnxRuntime.OnnxRuntimeException
HResult=0x80131500
Message=[ErrorCode:RuntimeException] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1103 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\Projects...\bin\Debug\net7.0-windows10.0.19041.0\win10-x64\AppX\onnxruntime_providers_cuda.dll"

Source=Microsoft.ML.OnnxRuntime
StackTrace:
at Microsoft.ML.OnnxRuntime.NativeApiStatus.VerifySuccess(IntPtr nativeStatus)
at Microsoft.ML.OnnxRuntime.SessionOptions.AppendExecutionProvider_CUDA(Int32 deviceId)
at Microsoft.ML.OnnxRuntime.SessionOptions.MakeSessionOptionWithCudaProvider(Int32 deviceId)
at VitCon.RemoveBackground.App.MauiProgram.CreateMauiApp() in D:\Projects\RemoveBackground\VitCon.RemoveBackground.App\MauiProgram.cs:line 13
at VitCon.RemoveBackground.App.WinUI.App.CreateMauiApp() in D:\Projects\RemoveBackground\VitCon.RemoveBackground.App\Platforms\Windows\App.xaml.cs:line 22
at Microsoft.Maui.MauiWinUIApplication.OnLaunched(LaunchActivatedEventArgs args)
at Microsoft.UI.Xaml.Application.Microsoft.UI.Xaml.IApplicationOverrides.OnLaunched(LaunchActivatedEventArgs args)
at ABI.Microsoft.UI.Xaml.IApplicationOverrides.Do_Abi_OnLaunched_0(IntPtr thisPtr, IntPtr args)

[mattleibow]

@thuongmhh let us know if you have a WinUI sample project or any other information.

[lhak]

I have seen such issues when a packaged app includes a dll with third-party dependencies. In this case, all of these dependencies also have to be included in the package/output directory.

[thuongmhh]

@lhak OnnxRuntime GPU version requires CUDA Dependency. If I understand correctly, I must copy CUDA dll files to the output directory, even when they are already in my PATH environment variable?

Is there any way for WinUI app to search for dlls that exist in PATH environment variable first, before the output directory?

[thuongmhh]

I found a solution. It seems that WinUI app only allows accessing dlls that are packaged in the output directory. So, I copied CUDA required libraries including cublas, cublasLt, cudart, cudnn, cufft, cufftw, curand, zlib, and OnnxRuntime GPU finally works.

[DrusTheAxe]

@lhak OnnxRuntime GPU version requires CUDA Dependency. If I understand correctly, I must copy CUDA dll files to the output directory, even when they are already in my PATH environment variable?

Is there any way for WinUI app to search for dlls that exist in PATH environment variable first, before the output directory?

PATH is not in the DLL Search Order for packaged apps. This is ByDesign.

For more details see https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order#search-order-for-packaged-apps

The recommended solution is an MSIX framework package containing CUDA, and then your package declares a <PackageDependency Name="CUDA"...> to include it in your process' package graph.

Alternatively, if you're a packaged or unpackaged app you can use Windows' Dynamic Dependency API on Win11 instead of declaring a static dependency in appxmanifest.xml.

Alternatively, you can include CUDA with your app as a SelfContained sort of solution.

(CUDA's license permitting, of course)