search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
13.44k stars 2.75k forks source link

Microsoft.ML.OnnxRuntime.Gpu not working in MAUI project #14974

Open thuongmhh opened 1 year ago

thuongmhh commented 1 year ago

Describe the issue

SessionOptions.MakeSessionOptionWithCudaProvider(0) raises OnnxRuntimeException: LoadLibrary failed with error 126.

I've tried targeting both .NET 6 and .NET 7, both of them produce the same exception.

The exception is gone when I call MakeSessionOptionWithCudaProvider in a console application, so I guess the problem is the integration between OnnxRuntime.GPU and MAUI only?

Here is the detail of the exception: Microsoft.ML.OnnxRuntime.OnnxRuntimeException HResult=0x80131500 Message=[ErrorCode:RuntimeException] D:\a_work\1\s\onnxruntime\core\session\provider_bridge_ort.cc:1103 onnxruntime::ProviderLibrary::Get [ONNXRuntimeError] : 1 : FAIL : LoadLibrary failed with error 126 "" when trying to load "D:\Projects...\bin\Debug\net7.0-windows10.0.19041.0\win10-x64\AppX\onnxruntime_providers_cuda.dll"

Source=Microsoft.ML.OnnxRuntime StackTrace: at Microsoft.ML.OnnxRuntime.NativeApiStatus.VerifySuccess(IntPtr nativeStatus) at Microsoft.ML.OnnxRuntime.SessionOptions.AppendExecutionProvider_CUDA(Int32 deviceId) at Microsoft.ML.OnnxRuntime.SessionOptions.MakeSessionOptionWithCudaProvider(Int32 deviceId) at VitCon.RemoveBackground.App.MauiProgram.CreateMauiApp() in D:\Projects\RemoveBackground\VitCon.RemoveBackground.App\MauiProgram.cs:line 13 at VitCon.RemoveBackground.App.WinUI.App.CreateMauiApp() in D:\Projects\RemoveBackground\VitCon.RemoveBackground.App\Platforms\Windows\App.xaml.cs:line 22 at Microsoft.Maui.MauiWinUIApplication.OnLaunched(LaunchActivatedEventArgs args) at Microsoft.UI.Xaml.Application.Microsoft.UI.Xaml.IApplicationOverrides.OnLaunched(LaunchActivatedEventArgs args) at ABI.Microsoft.UI.Xaml.IApplicationOverrides.Do_Abi_OnLaunched_0(IntPtr thisPtr, IntPtr args)

To reproduce

  1. Call SessionOptions.MakeSessionOptionWithCudaProvider(0) in a console application, and it should work well.
  2. Call SessionOptions.MakeSessionOptionWithCudaProvider(0) in a MAUI project, it should raise the OnnxRuntimeException exception.

Urgency

This issue is urgent. I'm developing a MAUI app that uses the ONNX Runtime library, and I can only do inference with CPU. I need this issue to be fixed so I can do inference with GPU.

Platform

Windows

OS Version

10.0.22621

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.14.0

ONNX Runtime API

C#

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.6

yuslepukhin commented 1 year ago

See https://github.com/microsoft/onnxruntime/issues/13658 first.

We typically ship onnxruntime_providers_cuda.dll, so if `onnxruntime.dll can be found then that one should be found too. I am not familiar with MAUI dll search order, but it seems to me that CUDA libraries can not be found from the app.

thuongmhh commented 1 year ago

I checked the output folder, both onnxruntime.dll and onnxruntime_providers_cuda.dll are there. How can I dig into the issue "LoadLibrary failed with error 126" to see what really happen?

AVISIX commented 1 year ago

I'm having the same issue with WPF (Windows Desktop). All the DLL's are there. I tried to call LoadLibrary manually, but even that failed. (Same error code: 126) So this is, probably, not an issue with onnx but instead with the DLL itself. Please let me know if theres an solution, I've been banging my head against a wall all day.

AVISIX commented 1 year ago

I'm having the same issue with WPF (Windows Desktop). All the DLL's are there. I tried to call LoadLibrary manually, but even that failed. (Same error code: 126) So this is, probably, not an issue with onnx but instead with the DLL itself. Please let me know if theres an solution, I've been banging my head against a wall all day.

Okay quick update. I was able to get it running after copying all the binaries from:

into the directory where all my project's binaries are (in my case, the debug folder).

I tested a bit further and it turns out its really just an issue with the PATH environment variable.

For the ones that are still struggeling, you can try the following:

If you want to share your software with others, you will have to add the following binaries to your project so they get copied: image

Hope this helps somewhat :)

Eric-Fung commented 7 months ago

I'm having the same issue with WPF (Windows Desktop). All the DLL's are there. I tried to call LoadLibrary manually, but even that failed. (Same error code: 126) So this is, probably, not an issue with onnx but instead with the DLL itself. Please let me know if theres an solution, I've been banging my head against a wall all day.

Okay quick update. I was able to get it running after copying all the binaries from:

  • CUDA (Version 11.2)
  • cudNN (Version 11 --> Binary version: 8.9.1.23) (+ the zlib dll)

into the directory where all my project's binaries are (in my case, the debug folder).

I tested a bit further and it turns out its really just an issue with the PATH environment variable.

For the ones that are still struggeling, you can try the following:

If you want to share your software with others, you will have to add the following binaries to your project so they get copied: image

Hope this helps somewhat :)

@AVISIX Hello AVISIX, my CUDA version is 12.2, which differs from yours, and my operating system is Windows 11. I've attempted the method you mentioned, but unfortunately, it didn't work. I believe the version difference should not significantly impact the results. It's possible that I made some mistakes in certain steps. Specifically:

andreasimeonegroovejones commented 4 months ago

I'm also having this issue with a simple Console app in net8.

Tried all that has been suggested in this thread but the problem still persist.

It is so strange to see that error as the .dll is exactly where the error says it cannot be found...

My setup:

Super frustrating...

skottmckay commented 3 months ago

I'm not sure it makes sense to use CUDA in a MAUI app. On a Windows machine you have to make sure your user has all CUDA binaries correctly installed and accessible to LoadLibrary (e.g. the user's PATH includes the necessary directories), including cuDNN binaries and the zlib dll. That's most likely over 2GB worth of files.

See https://github.com/microsoft/onnxruntime/issues/14974#issuecomment-1555890086 or https://github.com/microsoft/onnxruntime/issues/11092 for details.

Also note the CUDA/cuDNN install instructions frequently change. cuDNN seems to oscillate between adding binaries to the CUDA install and putting them in a separate directory. Due to that you need to follow the specific instructions for the CUDA/cuDNN versions you're attempting to use.

Alter-Nimbus commented 1 month ago

sometimes i feel like all those crappy installs are made on purpose to waste your time, always wondered why there is no tool for doing all those installs right on and with what you need... to do check ups and not waste your life and hours for looking just to paste files of a .lib like you do with cpp header mapping.