search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.87k stars 2.94k forks source link

Cannot run inference on Integrated Graphics with OpenVino EP using C Sharp API #13772

Open kimik-lu opened 2 years ago

kimik-lu commented 2 years ago

Describe the issue

I tried to run this C#-OpenVino EP example (https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_sharp/OpenVINO_EP/yolov3_object_detection) on the Intel® Integrated Graphics. The example application can run, but from the graph of the GPU/CPU performance on Task Manager, the Integrated Graphics is not used for inference. When running the example, the CPU usage is up to 100% (the peak in the CPU graph) and GPU usage remains nearly unchanged. image

To reproduce

  1. Build the NuGet packages of onnxruntime with openvino flavour: .\build.bat --config Release --use_openvino GPU_FP16 --build_shared_lib --build_nuget (the source code is OpenVINO EP v4.2 Release for ONNX Runtime & OpenVINO 2022.2 https://github.com/intel/onnxruntime/releases/tag/v4.2)
  2. Build the sample application and run the sample referencing the instruction: https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_sharp/OpenVINO_EP/yolov3_object_detection#build-the-sample-c-application (I was using Visual Studio 2022 and .Net 7.0, and also updated the source code at line 111 in program.cs from options.AppendExecutionProvider_OpenVINO(@"MYRIAD_FP16"); to options.AppendExecutionProvider_OpenVINO(@"GPU_FP16"); in order to use Integrated Graphics)

Expected behavior I expected to see the GPU usage in Task Manager to be higher when the sample was doing inference while CPU usage to be lower.

Urgency

No response

Platform

Windows

OS Version

Windows 10

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.13.0

ONNX Runtime API

C#

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

OpenVINO EP: v4.2; OpenVINO: 2022.2

jywu-msft commented 2 years ago

@sfatimar , any ideas?

sfatimar commented 2 years ago

This was working fine. Please download the nuget package directly from . https://github.com/intel/onnxruntime/releases/download/v4.2/nuget-artifacts-windows-v4.2.zip and check once,

kimik-lu commented 2 years ago

This was working fine. Please download the nuget package directly from . https://github.com/intel/onnxruntime/releases/download/v4.2/nuget-artifacts-windows-v4.2.zip and check once,

Thanks for your information. I changed to use the NuGet packages from the link you provided, and the GPU usage didn't seem to be increased when inference. Then I downgraded the driver of the graphics from 30.0.101.1340 to 27.20.100.9079, it looks like the integrated graphics get engaged when inference. Another thing to note is that with the graphic driver 30.0 and the raw OpenVino C++ API, the GPU can be used for inference. So, I was wondering if this version of ORT+OpenVino EP does not support the driver 30.0?

sfatimar commented 2 years ago

GPU driver version available in DG2 Windows machine is 30.0.101.1660. GPU is being utilized. Downloaded Nuget packages from https://github.com/intel/onnxruntime/releases/download/v4.2/nuget-artifacts-windows-v4.2.zip and followed commands mentioned as below mkdir source mkdir val cd val dotnet new console Copy the scripts from https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_sharp/OpenVINO_EP/yolov3_object_detection to val directory in program.cs file, modify line 111 MYRIAD_FP16 to GPU_FP16 dotnet add package SixLabors.ImageSharp dotnet add package SixLabors.Core --version 1.0.0-beta0008 dotnet add package SixLabors.Fonts --version 1.0.0-beta19 dotnet add package SixLabors.ImageSharp.Drawing --version 1.0.0-beta15

nuget add C:\Users\mohsinmx\Downloads\nuget-artifacts-windows-v4.2\nuget-artifacts-windows-v4.2\Microsoft.ML.OnnxRuntime.OpenVino.1.13.1-dev-20221026-1209-861125ccb.nupkg -Source C:\Users\mohsinmx\Downloads\mohsin-nugget\source\

nuget add C:\Users\mohsinmx\Downloads\nuget-artifacts-windows-v4.2\nuget-artifacts-windows-v4.2\Microsoft.ML.OnnxRuntime.Managed.1.13.1-dev-20221026-1209-861125ccb.nupkg -Source C:\Users\mohsinmx\Downloads\mohsin-nugget\source\

nuget sources Add -Name "cSharpSample" -Source C:\Users\mohsinmx\Downloads\mohsin-nugget\source

dotnet add package Microsoft.ML.OnnxRuntime.OpenVino -v 1.13.1-dev-20221026-1209-861125ccb -s C:\Users\mohsinmx\Downloads\mohsin-nugget\source

dotnet add package Microsoft.ML.OnnxRuntime.Managed -v 1.13.1-dev-20221026-1209-861125ccb -s C:\Users\mohsinmx\Downloads\mohsin-nugget\source

dotnet build

dotnet run C:\Users\mohsinmx\Downloads\OD_FP32\yolov3\yolov3-12.onnx C:\Users\mohsinmx\Downloads\onnxruntime-inference-examples-main\onnxruntime-inference-examples-main\quantization\notebooks\imagenet_v2\calibration_imagenet\cat.jpg C:\Users\mohsinmx\Downloads\output2.bmp

kimik-lu commented 1 year ago

GPU driver version available in DG2 Windows machine is 30.0.101.1660. GPU is being utilized. Downloaded Nuget packages from https://github.com/intel/onnxruntime/releases/download/v4.2/nuget-artifacts-windows-v4.2.zip and followed commands mentioned as below mkdir source mkdir val cd val dotnet new console Copy the scripts from https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_sharp/OpenVINO_EP/yolov3_object_detection to val directory in program.cs file, modify line 111 MYRIAD_FP16 to GPU_FP16 dotnet add package SixLabors.ImageSharp dotnet add package SixLabors.Core --version 1.0.0-beta0008 dotnet add package SixLabors.Fonts --version 1.0.0-beta19 dotnet add package SixLabors.ImageSharp.Drawing --version 1.0.0-beta15

nuget add C:\Users\mohsinmx\Downloads\nuget-artifacts-windows-v4.2\nuget-artifacts-windows-v4.2\Microsoft.ML.OnnxRuntime.OpenVino.1.13.1-dev-20221026-1209-861125ccb.nupkg -Source C:\Users\mohsinmx\Downloads\mohsin-nugget\source\

nuget add C:\Users\mohsinmx\Downloads\nuget-artifacts-windows-v4.2\nuget-artifacts-windows-v4.2\Microsoft.ML.OnnxRuntime.Managed.1.13.1-dev-20221026-1209-861125ccb.nupkg -Source C:\Users\mohsinmx\Downloads\mohsin-nugget\source\

nuget sources Add -Name "cSharpSample" -Source C:\Users\mohsinmx\Downloads\mohsin-nugget\source

dotnet add package Microsoft.ML.OnnxRuntime.OpenVino -v 1.13.1-dev-20221026-1209-861125ccb -s C:\Users\mohsinmx\Downloads\mohsin-nugget\source

dotnet add package Microsoft.ML.OnnxRuntime.Managed -v 1.13.1-dev-20221026-1209-861125ccb -s C:\Users\mohsinmx\Downloads\mohsin-nugget\source

dotnet build

dotnet run C:\Users\mohsinmx\Downloads\OD_FP32\yolov3\yolov3-12.onnx C:\Users\mohsinmx\Downloads\onnxruntime-inference-examples-main\onnxruntime-inference-examples-main\quantization\notebooks\imagenet_v2\calibration_imagenet\cat.jpg C:\Users\mohsinmx\Downloads\output2.bmp

Thanks for your detailed guidance. After I reinstall the same version of GPU driver, it looks like it works now. Thank you very much!