search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.46k stars 2.9k forks source link

Can I use Microsoft.AI.MachineLearning(WinML) interfaces to enable OnnxRuntime-QNN-EP(Qualcomm NPU) or other no-DirectX EP ? #19361

Open caofx0418 opened 8 months ago

caofx0418 commented 8 months ago

Describe the feature request

Can I use Microsoft.AI.MachineLearning(WinML) interfaces to enable OnnxRuntime-QNN-EP(Qualcomm NPU) or other no-DirectX EP ?

  1. From "windows-ml/get-started", we can see 2 solutions: In-box vs NuGet https://learn.microsoft.com/en-us/windows/ai/windows-ml/get-started

  2. In-box: OnnxRuntime as a part of Windows, we can't change. so it's no chance to use myself built OnnxRuntime with QNN-EP. image

  3. NuGet: Microsoft.ai.machinelearning.dll does not contain an embedded ONNX runtime, instead the ONNX runtime is built into a file: onnxruntime.dll. image

    and maybe we can replace the onnxruntime.dll to myself built OnnxRuntime with QNN-EP. but the WinML interface: public enum LearningModelDeviceKind, only support 4 kinds: 0 Default, 1 Cpu, 2 DirectX, 3 DirectXHighPerformance, 4 DirectXMinPower. But there is no option for QNN-EP or other EP in the LearningModelDeviceKind enumeration.

So can I use Microsoft.AI.MachineLearning(WinML) interfaces to enable OnnxRuntime-QNN-EP(Qualcomm NPU) or other no-DirectX EP?

Describe scenario use case

Windows: Microsoft.AI.MachineLearning(WinML) interfaces to enable OnnxRuntime-QNN-EP(Qualcomm NPU) or other no-DirectX EP

caofx0418 commented 8 months ago

@smk2007

Can you help me? Thank you.

caofx0418 commented 8 months ago

@smk2007 @PatriceVignola @justinchuby @snnn @HectorSVC could you please take a look this issue? thank you

caofx0418 commented 8 months ago

@jywu-msft @fdwr

could you please take a look this issue? thank you

fdwr commented 8 months ago

Can I use WinML ... to enable QNN-EP...?

@caofx0418 Nope, WinML only supports CPU and DirectX (DML) backends.

https://github.com/microsoft/onnxruntime/blob/61e07a46e17c6dbc1adbd824d6b36aaac6f26c7b/winml/lib/Api/LearningModelDevice.cpp#L27-L35

caofx0418 commented 8 months ago

Can I use WinML ... to enable QNN-EP...?

@caofx0418 Nope, WinML only supports CPU and DirectX (DML) backends.

https://github.com/microsoft/onnxruntime/blob/61e07a46e17c6dbc1adbd824d6b36aaac6f26c7b/winml/lib/Api/LearningModelDevice.cpp#L27-L35

Thank you for your reply!

Do WinML have plan to support other Backends in future ？

fdwr commented 8 months ago

Do WinML have plan to support other Backends in future ？

I defer to Sheil @smk2007 for any future WinML plans, but my wager is that WinML will only target Windows API's like DirectML rather than using lower-level device-specific driver interfaces. WinML is a thin layer over ORT though - can you call ORT directly?

caofx0418 commented 8 months ago

Do WinML have plan to support other Backends in future ？

I defer to Sheil @smk2007 for any future WinML plans, but my wager is that WinML will only target Windows API's like DirectML rather than using lower-level device-specific driver interfaces. WinML is a thin layer over ORT though - can you call ORT directly?

Thank you for your reply!

I can call ORT C++ interface directly. and I also want to make my EP more "Windows ML native".