search

microsoft / onnxruntime-inference-examples

Examples for using ONNX Runtime for machine learning inferencing.
MIT License
1.16k stars 331 forks source link

[mobile] Move to net8 MAUI and add MacCatalyst support #406

Open rmarinho opened 6 months ago

rmarinho commented 6 months ago

This pull request primarily aims to update the MauiVisionSample project in the mobile/examples/Maui directory to support the MacCatalyst platform, along with some other improvements and updates. The most significant changes include updating the target frameworks in the project file, adding MacCatalyst-specific files, updating package references, and adding debug logging.

This pull request primarily focuses on updating the MauiVisionSample application to support the MacCatalyst platform and updating the target frameworks to .NET 8.0. It also includes changes to improve debugging, and package updates.

The most important changes are:

This pull request includes several changes to enhance compatibility with .NET 8.0, add support for MacCatalyst, and update dependencies in the MauiVisionSample and MauiSuperResolution projects.

Compatibility and Platform Support:

Dependency Updates:

Logging and Debugging:

MacCatalyst Specific Files:

Solution Files:

rmarinho commented 6 months ago

Right now we are hitting this on Catalyst 

/usr/local/share/dotnet/packs/Microsoft.MacCatalyst.Sdk/17.2.8043/targets/Xamarin.Shared.Sdk.targets(3,3): Error: No matching framework found inside '/Users/ruimarinho/.nuget/packages/microsoft.ml.onnxruntime.extensions.dummy/0.10.0/runtimes/ios/native/onnxruntime_extensions.xcframework'. SupportedPlatform: '/Users/ruimarinho/.nuget/packages/microsoft.ml.onnxruntime.extensions.dummy/0.10.0/runtimes/ios/native/onnxruntime_extensions.xcframework', SupportedPlatformVariant: 'ios', SupportedArchitectures: 'maccatalyst'. (MauiVisionSample)

I have talked with @YUNQIUGUO and we identified the issue , we need to update the Microsoft.ML.Onnxruntime.Extensions.Dummy package to support MacCatalyst too for x86 and arm

YUNQIUGUO commented 5 months ago

Update: We identified the issue that for this to work, the onnxruntime_extensions.xcframework requires catalyst build support to be added as well. And with that supported, @rmarinho helped verify with a Nuget ORT package contains catalyst support and Nuget ORT Extension package with catalyst support (manually replaced and tested for now), this .NET MAUI app works in MacCatalyst platform.

So next step is when the official/dev version dummy extensions nuget package is released, we can update the package dependency here in this project and should be good to go.

rmarinho commented 5 months ago

Yap sounds good . will wait for dev package

rmarinho commented 3 months ago

Hello, I just tested the sample with the latest version of the onxx runtime and seems to be working fine, can we merge this?

Thanks

rmarinho commented 2 months ago

@YUNQIUGUO do you think this can be merged?

lutzapps commented 1 month ago

@rmarinho & @YUNQIUGUO I think there is still a problem with the "net8.0-maccatalyst" (x64) target support for the "Microsoft.ML.OnnxRuntime.Extensions", and I mean with the full Runtime Extensions and not the "Microsoft.ML.OnnxRuntime.Extensions.Dummy" nugget (which has no real implementation, and is only needed to fullfill iOS build requirement).

With the latest prerelease version "0.11.0-dev-20240524-2333-8d8670f7" of "Microsoft.ML.OnnxRuntime.Extensions", I was able to run the "MAUISuperResolution" perfectly fine for the Android target and also all fine in the iOS-Simulator. But when running the same code with MacCatalyst x64 target, I get an runtime error at inference time:

[ErrorCode:Fail] Fatal error: com.microsoft.extensions:DecodeImage(-1) is not a registered function/op

In contrast to the "MauiVisionSample" which does NOT use a pre-/postprocessing pipeline in the ONNX model, the "MauiSuperResolution" sample DOES USE and need the Extensions support for custom pre-/post processing like DecodeImage() and EncodeImage() Vision Operators:

const std::vector<const OrtCustomOp*>& VisionLoader() {
  static OrtOpLoader op_loader(CustomCpuStruct("EncodeImage", ort_extensions::KernelEncodeImage),
                               CustomCpuStruct("DecodeImage", ort_extensions::KernelDecodeImage),
                               CustomCpuStruct("DrawBoundingBoxes", ort_extensions::DrawBoundingBoxes));
  return op_loader.GetCustomOps();
}

FxLoadCustomOpFactory LoadCustomOpClasses_Vision = VisionLoader;

But even as the code in "MauiSuperResolution" registers for the Runtime Extensions support with following code in "OrtInferenceSession.cs":

            // enable pre/post processing custom operators from onnxruntime-extensions
L37:       _sessionOptions.RegisterOrtExtensions();

that ORT extensions are only found at runtime for Android and iOS, but NOT for MacCatalyst-x64.

I checked the "bin/debug/net8-maccatalyst/maccatalyst-x64" directory, and there is a "onyxruntime_extensions" Unix Executable File, but somehow that is not found at runtime or it is the wrong version for maccatalyst-x64.

So I think it is not sufficient to test ORT Extensions support or .NET8 support with the "MauiVisionSample", which only needs the Dummy Extensions. The test should be done with "MauiSuperResolution", which DOES use (and need) the ONNX Runtime Extensions. The latest "0.11.0-dev-20240524-2333-8d8670f7" nugget package of "Microsoft.ML.OnnxRuntime.Extensions" now allows to compile without error against the "net8.0-maccatalyst" target, but it can not find the standard ONNX Runtime Extensions functions/ops at runtime at inference time.

Verified with: macOS Sonoma 14.6 on MacBookPro (Intel x64) OS: Darwin x64 23.6.0 Xcode-15.4.0 VSCode Version: 1.92.1 (Universal) .NET SDK 8.0.303 .NET MAUI workload:: Installation Source: SDK 8.0.300, Manifest Version: 8.0.61/8.0.100

MauiSuperResolution.csproj::

<TargetFrameworks>net8.0-android;net8.0-ios;net8.0-maccatalyst</TargetFrameworks>

<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'ios'">14.2</SupportedOSPlatformVersion>
<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'maccatalyst'">14.0</SupportedOSPlatformVersion>
<SupportedOSPlatformVersion Condition="$([MSBuild]::GetTargetPlatformIdentifier('$(TargetFramework)')) == 'android'">21.0</SupportedOSPlatformVersion>

<PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="8.0.0" />
<PackageReference Include="Microsoft.Maui.Controls" Version="$(MauiVersion)" />
<PackageReference Include="Microsoft.Maui.Controls.Compatibility" Version="$(MauiVersion)" />
<PackageReference Include="Microsoft.ML.OnnxRuntime" Version="1.18.1" />
<PackageReference Include="Microsoft.ML.OnnxRuntime.Extensions" Version="0.11.0-dev-20240524-2333-8d8670f7" />
<PackageReference Include="Microsoft.ML.OnnxRuntime.Managed" Version="1.18.1" />
skottmckay commented 1 month ago

The latest release of the extensions package should now have a mac-catalyst build.

https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.Extensions/0.12.0

rmarinho commented 4 weeks ago

yap I updated all packages and I also hit that error on MacCatalyst on MBP M3 using the MAUISuperResolution" wi

Screenshot 2024-09-08 at 01 57 31

I do think we can merge this pr as it fixes and updates most for net8

 Documents dotnet --info
.NET SDK:
 Version:           9.0.100-rc.1.24452.12
 Commit:            81a714c6d3
 Workload version:  9.0.100-rc.1.24453.3
 MSBuild version:   17.12.0-preview-24422-09+d17ec720d

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  14.6
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/9.0.100-rc.1.24452.12/

.NET workloads installed:
Configured to use workload sets when installing new manifests.
 [maui]
   Installation Source: SDK 9.0.100-rc.1
   Manifest Version:    9.0.0-rc.1.24453.9/9.0.100-rc.1
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/9.0.100-rc.1/microsoft.net.sdk.maui/9.0.0-rc.1.24453.9/WorkloadManifest.json
   Install Type:        FileBased

Host:
  Version:      9.0.0-rc.1.24431.7
  Architecture: arm64
  Commit:       static

.NET SDKs installed:
  7.0.317 [/usr/local/share/dotnet/sdk]
  8.0.401 [/usr/local/share/dotnet/sdk]
  9.0.100-rc.1.24452.12 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.20 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0-rc.1.24452.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.20 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0-rc.1.24431.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download
lutzapps commented 4 weeks ago

@skottmckay and @rmarinho The problem I have is NOT that I not get a valid build for Mac Catalyst, but that an App which actually really then USES the ORT-Extensions errors out at runtime, because of something is still missing.

Now with version 0.12 the error changed to "One or more errors occurred. (The type initializer for 'Microsoft.ML.OnnxRuntime.NativeMethods' threw an exception.)".

With v 0.11-dev it was another error I send here before/above. So with your example of the "Dummy" package it may be possible now to built for MacCatalyst, but when you actually use ORT-Extension v0.12 it does NOT work on MacCatalyst.

skottmckay commented 3 weeks ago

Unfortunately the build config for mac-catalyst is excluding the image decoding/encoding operators due to an issue with opencv. This PR should hopefully address that.

rmarinho commented 1 week ago

Any else I need to do to get this merged? Thanks

lutzapps commented 1 week ago

@skottmckay so basically your mentioned PR only is a different wording for a clearer error message like "NSImage is not supported on Mac Catalyst", but the image decoding/encoding operators will NOT be included for Mac Catalyst? Is there no hope that this issue with opencv can be fixed for Mac Catalyst?