Onnxruntime NuGet package throw an exception when call "AppendExecutionProvider_Nnapi"

[shingming]

Is your feature request related to a problem? Please describe. I'm using onnxruntime NuGet package (C#) with Unity. When I build an app for android, it always throws an exception when calling "AppendExecutionProvider_Nnapi". The information about the exception is "The NNAPI Execution Provider is not supported in this build". Have any schedule for implementing the feature about calling "AppendExecutionProvider_Nnapi" API in C#?

System information Platform: Android ONNX Runtime version: Microsoft.ML.OnnxRuntime.Gpu 1.11.0 Unity version: 2021.2.18f1

Describe the solution you'd like Implement the feature about calling "AppendExecutionProvider_Nnapi" API in C#

Additional context

[skottmckay]

It's supported in the CPU package - Microsoft.ML.OnnxRuntime.

The GPU package is extremely large so most likely isn't a good fit if you're targeting Android devices.

[shingming]

@skottmckay thank you for your info. My model can infer by the CPU. If not consider the size, is there tutorial for building .dll which include NNAPI implementation?

[shingming]

@skottmckay Sorry, I think I misunderstand your reply. Is there NNAPI implementation in "Microsoft.ML.OnnxRuntime"? And due to the size, it is not included in "Microsoft.ML.OnnxRuntime.GPU" right?

So, can I access the GPU resources and infer the model using GPU on Android through Microsoft.ML.OnnxRuntime?

[skottmckay]

The NNAPI execution provider is small. The GPU package is large due to it containing CUDA kernels. Note that it is CUDA specific and not a generic package that works with any GPU. I'm not sure how many Android devices support CUDA either.

Because the GPU package is large, it is not likely someone would want to use that on a mobile device, hence we only include the NNAPI execution provider in the much smaller CPU package.

The ORT NNAPI execution provider converts the ONNX model to an NNAPI model at runtime, and the NNAPI implementation on the device executes that NNAPI model. The NNAPI implementation on the device can potentially use GPU or NPU, but that implementation is device specific and varies by manufacturer. It also depends on the model as to how much of it can be run using NNAPI. The usability checker mentioned in https://onnxruntime.ai/docs/reference/mobile/helpers.html can give an idea about that.

[shingming]

@skottmckay

Got it, now I can see the implementation of "AppendExecutionProvider_Nnapi" API

Although I re-build the app using Unity Engine with the CPU package and then install on the VR device (Oculus Quest 2), I get the same error messages through ADB. I have no idea why.

[shingming]

I follow this guide to add (onnxruntime) packages from NuGet to a Unity project: https://docs.microsoft.com/en-us/visualstudio/gamedev/unity/unity-scripting-upgrade#add-packages-from-nuget-to-a-unity-project

The following image is the screenshot when download the onnxruntime CPU package:

I analyze the Microsoft.ML.OnnxRuntime.dll (downloaded at the previous step) using "Jetbrains DdotPeek" (.NET Decompiler & Assembly Browser). It seems like the CPU package also does not implement the method?!!!

[skottmckay]

There are different libraries for each platform in the managed nuget package under the lib directory. Are you looking at the one in lib\monoandroid11.0?

[shingming]

@skottmckay I see. I use netstand2.0 for building, so now I know why I get the same error with the CPU package. It is not included on the netstand2 platform. I check other building, and it is not included at all except monoandroid11.0.

I use monoandroid11.0 for the Unity building, and it throws the error when the Unity project settings are: Setting 1: Scripting Backend: Mono (also appear error when setting at IL2CPP) Api Compatibility Level: .NET Framework Throws error mess: ArgumentException: The Assembly Mono.Android is referenced by Microsoft.ML.OnnxRuntime (‘Assets/Plugins/Microsoft.ML.OnnxRuntime.dll’). But the dll is not allowed to be included or could not be found.

Setting 2: Scripting Backend: Mono (also appear error when setting at IL2CPP) Api Compatibility Level: .NET Standard 2.1 Throws error mess: error CS7069: Reference to type 'ReadOnlySpan<>' claims it is defined in 'mscorlib', but it could not be found (the corresponding line code: ArrayTensorExtensions.ToTensor)

I think the .dll build with monoandroid11.0 is not supported on building using Unity Engine. Would the team consider building “nnapi” feature on (Microsoft.ML.OnnxRuntime.dll) netstand2.0? Because Unity can build correctly using (Microsoft.ML.OnnxRuntime.dll) netstand2.0 version (but throw the not implementation exception now). I think the community has some people like me who are building Android apps by Unity Engine.

[skottmckay]

AFAIK it's not possible to make it work with just a netstandard2.0 target as we need some extra attributes that are provided by Xamarin/.net6 to make calling into the native C++ library work. The monoandroid target framework is provided by Xamarin.

e.g. https://github.com/microsoft/onnxruntime/blob/75cf5dc2c9260e082172ea4e8c5384aef6e4cfe2/csharp/src/Microsoft.ML.OnnxRuntime/AssemblyInfo.shared.cs#L1-L14

[shingming]

@skottmckay ok, It seems like I must use monoandroid for using nnapi on android.

Btw, the following message is about the error code, I think the error when building apps with monoandroid11.0 is caused by the side of Unity Engine. I will post a question on the Unity forum. Thank you for your information :)

Building Library\Bee\artifacts\Android\ManagedStripped failed with output: C:\Program Files\Unity\Hub\Editor\2021.2.18f1\Editor\Data\il2cpp\build\deploy\UnityLinker.exe --search-directory=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed --out=Library/Bee/artifacts/Android/ManagedStripped --include-link-xml=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed\MethodsToPreserve.xml --include-link-xml=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed\TypesInScenes.xml --include-link-xml=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed\SerializedTypes.xml --include-link-xml=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Assets..\Temp\InputSystemLink.xml --include-link-xml=C:\Users\timla\Documents\GitHub\MetaverseProject\VirtualEnvironmentGeneration\Temp\burst.link.xml --include-link-xml=C:\Users\timla\Documents\GitHub\MetaverseProject\VirtualEnvironmentGeneration\Assets\Photon\PhotonUnityNetworking\link.xml --include-link-xml=C:/Program Files/Unity/Hub/Editor/2021.2.18f1/Editor/Data/PlaybackEngines/AndroidPlayer/Tools/AndroidNativeLink.xml --include-directory=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed --dotnetprofile=unityaot-linux --dotnetruntime=Il2Cpp --platform=Android --use-editor-options --editor-settings-flag=None,Development --enable-engine-module-stripping --engine-modules-asset-file=C:/Program Files/Unity/Hub/Editor/2021.2.18f1/Editor/Data/PlaybackEngines/AndroidPlayer/modules.asset --editor-data-file=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/EditorToUnityLinkerData.json --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Assembly-CSharp-firstpass.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Assembly-CSharp.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/PhotonChat.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Oculus.Voice.Demo.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Oculus.AudioManager.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Oculus.Interaction.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/AssistantCoreSDKRuntime.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Unity.InputSystem.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/UnityEngine.UI.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Facebook.WitAI.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/PhotonUnityNetworking.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Oculus.VR.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Klak.Ndi.Runtime.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Unity.XR.Management.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/PhotonVoice.PUN.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Oculus.Interaction.Samples.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Unity.Barracuda.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Oculus.Spatializer.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/PhotonVoice.API.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Unity.TextMeshPro.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Unity.XR.Interaction.Toolkit.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Unity.RenderPipelines.Core.Runtime.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Paroxe.PDFRenderer.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/PhotonRealtime.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/PhotonUnityNetworking.Utilities.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/PhotonVoice.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Oculus.Platform.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/PhotonUnityNetworking.Demos.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Oculus.LipSync.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/VoiceSDK.Runtime.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/Unity.XR.Oculus.dll --include-unity-root-assembly=C:/Users/timla/Documents/GitHub/MetaverseProject/VirtualEnvironmentGeneration/Temp/StagingArea/Data/Managed/DOTween.dll --print-command-line Fatal error in Unity CIL Linker Mono.Cecil.AssemblyResolutionException: Failed to resolve assembly: 'Mono.Android, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' at Unity.IL2CPP.Common.MissingMethodStubber.GetTypeModule(TypeReference type, IEnumerable' 1 assemblies) at Unity.Linker.Steps.AddUnresolvedStubsStep.MarkAssemblyOfType(UnityLinkContext context, TypeReference type) at Unity.Linker.Steps.Marking.UnresolvedStubMarking.HandleUnresolvedType(TypeReference reference) at Unity.Linker.Steps.Marking.UnresolvedStubMarking.HandleUnresolvedMethod(MethodReference reference) at Unity.Linker.Steps.UnityMarkStep.HandleUnresolvedMethod(MethodReference reference) at Mono.Linker.Steps.MarkStep.ProcessLazyAttributes() at Mono.Linker.Steps.MarkStep.Process() at Mono.Linker.Steps.MarkStep.Process(LinkContext context) at Unity.Linker.Steps.UnityMarkStep.Process(LinkContext context) at Unity.Linker.UnityPipeline.ProcessStep(LinkContext context, IStep step) at Mono.Linker.Pipeline.Process(LinkContext context) at Unity.Linker.UnityDriver.UnityRun(Boolean noProfilerAllowed, ILogger customLogger) at Unity.Linker.UnityDriver.RunDriverWithoutErrorHandling(ILogger customLogger, Boolean noProfilerAllowed) at Unity.Linker.UnityDriver.RunDriver() UnityEngine.GUIUtility:ProcessEvent (int,intptr,bool&)

[skottmckay]

Based on this old issue it sounds like there are ways to target Android with Unity and the existing ORT package:

10427

[shingming]

@skottmckay It fix in Unity 2021.2.X(which I use) and the error message seems not related to this problem

[skottmckay]

Closing as ORT issue was resolved. Remaining issue seems to be due to Unity.

[zzbuzzard]

Hey @shingming, I've been having the same problem! Is there any chance you could describe how you fixed it or link to the unity forum where you discussed it? No matter what I do, I get the "The NNAPI Execution Provider is not supported in this build" exception when running on Android :(

Details: I'm using Microsoft.ML.OnnxRuntime 1.16.3 (and Microsoft.ML.OnnxRuntime.Managed 1.16.3). I've tried using both Unity 2021.3.x and 2022.3.x. My target API is level 30 (Android 11.0), I'm using Mono and .NET Standard 2.1, but also tried .NET Framework (no build errors but same problem). I have implementation 'com.microsoft.onnxruntime:onnxruntime-android:1.16.3' added to my gradle file.

Inference works fine without NNAPI.

Any clues? Is Unity somehow not using monoandroid11.0 for OnnxRuntime? How can I get more information about what's going on? Apologies if this is more of a Unity question than onnxruntime, I might also ask on a forum there.

EDIT: Also tried IL2CPP and downgrading to Microsoft.ML.OnnxRuntime version 1.11.0 but no dice ;(

[zzbuzzard]

Hey @skottmckay sorry to tag, but I think this is unresolved! I don't want to open a new issue as I believe I have an identical problem to above. Any pointers would be v appreciated.