search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.86k stars 2.94k forks source link

C Sharp API for openvino doesn't run on GPU #5011

Open jbrownkramer opened 4 years ago

jbrownkramer commented 4 years ago

Describe the bug I followed the instructions here: https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/OpenVINO-ExecutionProvider.md#csharp-api to build a nuget package to provide a c sharp API for the OpenVINO execution provider. More accurately, that failed, I opened issue #4945, and with some help and a modified build string, I created the nuget packages. There were more struggles to actually add them to my project, but finally I've got it doing inference. However, it is quite slow (much slower than directly using the openvino python API), and the GPU is not being engaged, despite built with --use_openvino GPU_FP16

Urgency We have a C# codebase and we want to do inference on an integrated Intel GPU. This is the only option we've found for doing this with C#.

System information

To Reproduce

            string onnxFilePath = ...
            SessionOptions bob = new SessionOptions();
            //bob.GraphOptimizationLevel = GraphOptimizationLevel.ORT_DISABLE_ALL;
            //bob.AppendExecutionProvider_OpenVINO("GPU_FP16");
            InferenceSession inferenceSession = new InferenceSession(onnxFilePath/*, bob*/);
            //etc

Expected behavior I expect inference time around 70ms on a resnet 50 with 512x424 images and to see activity on the GPU.

Additional context I tried various combinations of uncommenting the session session settings. Disabling optimizations is VERY slow. Using AppendExecutionProvider_OpenVINO gives the result

'Unable to find an entry point named 'OrtSessionOptionsAppendExecutionProvider_OpenVINO' in DLL 'onnxruntime'
jywu-msft commented 4 years ago

are you comparing performance of debug builds? and I assume on python all is fine (you see gpu activity there) and what version of OpenVINO is this being tested on ? adding some other folks for their input. +@smkarlap, @sfatimar

sfatimar commented 4 years ago

is there a file openvino_provider_factory.h in your nuget package ? If the file is not there then this error may occur 'Unable to find an entry point named 'OrtSessionOptionsAppendExecutionProvider_OpenVINO' in DLL 'onnxruntime' The latest method to build csharp is in https://github.com/microsoft/onnxruntime/blob/master/BUILD.md with the flag ---build_nuget and --build_shared_libs flag

jbrownkramer commented 4 years ago

@jywu-msft

are you comparing performance of debug builds? and I assume on python all is fine (you see gpu activity there) and what version of OpenVINO is this being tested on ? adding some other folks for their input. +@smkarlap, @sfatimar

When I use the openvino python api I see GPU activity, which presumably is why there is a speedup. This is OpenVino 2020.4.287

jbrownkramer commented 4 years ago

@sfatimar

is there a file openvino_provider_factory.h in your nuget package ? If the file is not there then this error may occur 'Unable to find an entry point named 'OrtSessionOptionsAppendExecutionProvider_OpenVINO' in DLL 'onnxruntime' The latest method to build csharp is in https://github.com/microsoft/onnxruntime/blob/master/BUILD.md with the flag ---build_nuget and --build_shared_libs flag

Yes. openvino_provider_factory.h is in my package. However, I think there is some issue with the resources from the package being properly deployed when I build my project. In particular, the onnxruntime.dll isn't in the build directory of my project unless I manually copy it there. Copying the header files into the build folder doesn't help.

sfatimar commented 4 years ago

I was able to build the latest version of onnxruntime with openvino using this command with vs studio command prompt SETUP OPENVINO ENVIRONMENT(program_files x86/IntelSwTools/openvino_2020.4.287/bin/setupvars.bat) /build.bat --config Debug --use_openvino CPU_FP32 --build --update --cmake_generator "Visual Studio 16 2019" --build_shared_lib --build_nuget with a small change in cmake/onnxruntime_providers.cmake to add /wd4267 warning (this is caused by openvino code )

target_compile_options(onnxruntime_providers_openvino PUBLIC /wd4275 /wd4100 /wd4005 /wd4244 /wd4267)

The nuget packages are created in the folder called nuget-artifacts. Originally we bought this feature C# only on Linux systems, hence the glitches...

jbrownkramer commented 4 years ago

@sfatimar, thank you for this. There is still a problem. Namely, I get this error:

2020-09-09 14:54:48,894 Build [DEBUG] - Running subprocess in 'C:\Users\josh\projects\onnxruntime\csharp'
['dotnet', 'restore', 'OnnxRuntime.CSharp.sln', '--configfile', 'Nuget.CSharp.config']
  Determining projects to restore...
C:\Program Files\dotnet\sdk\3.1.401\NuGet.targets(128,5): error : 'Python 3.8.4' is not a valid version string. (Parameter 'value') [C:\Users\josh\projects\onnxruntime\csharp\OnnxRuntime.CSharp.sln]
Traceback (most recent call last):
  File "C:\Users\josh\projects\onnxruntime\\tools\ci_build\build.py", line 1841, in <module>
    sys.exit(main())
  File "C:\Users\josh\projects\onnxruntime\\tools\ci_build\build.py", line 1817, in main
    build_nuget_package(
  File "C:\Users\josh\projects\onnxruntime\\tools\ci_build\build.py", line 1458, in build_nuget_package
    run_subprocess(cmd_args, cwd=build_dir)
  File "C:\Users\josh\projects\onnxruntime\\tools\ci_build\build.py", line 427, in run_subprocess
    completed_process = subprocess.run(
  File "C:\Users\josh\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dotnet', 'restore', 'OnnxRuntime.CSharp.sln', '--configfile', 'Nuget.CSharp.config']' returned non-zero exit status 1.

There seems to be some issue with python. I know that openvino says it supports python 3.5-3.7 in the windows install instructions. I tried it with a somewhat weird installation of python 3.6.5 as well and got the same errorerror : 'Python 3.6.5' is not a valid version string.

I'll try a more standard installation of python 3.6.5 and see if that works.

jbrownkramer commented 4 years ago

I installed python 3.6.5 from https://www.python.org/downloads/release/python-365/ with all the default settings, and added it to PATH. I verified that my installation of openvino worked, and tried the command line you suggested. Same problem.

C:\Program Files\dotnet\sdk\3.1.401\NuGet.targets(128,5): error : 'Python 3.6.5' is not a valid version string. (Parameter 'value') [C:\Users\josh\projects\onnxruntime\csharp\OnnxRuntime.CSharp.sln]

One note: I am using the current master branch of this repository.

jbrownkramer commented 4 years ago

Ok. I dug into it further. There is an environment variable called version set by setupvars.bat in openvino/bin. This is set to the python version, which, maybe, onnxruntime version is mistaking for a nuget version? At any rate, I deleted the lines in setupvars.bat that set this variable, and the nuget packages built.

What's more, when I added the packages to my project, it still isn't moving onnxruntime.dll into the binaries folder.

In fact, if I manually move the onnxruntime.dll into the build folder it STILL says Unable to load DLL 'onnxruntime'. I'm stuck here. Any suggestions?

yuslepukhin commented 4 years ago

Ok. I dug into it further. There is an environment variable called version set by setupvars.bat in openvino/bin. This is set to the python version, which, maybe, onnxruntime version is mistaking for a nuget version? At any rate, I deleted the lines in setupvars.bat that set this variable, and the nuget packages built.

What's more, when I added the packages to my project, it still isn't moving onnxruntime.dll into the binaries folder.

In fact, if I manually move the onnxruntime.dll into the build folder it STILL says Unable to load DLL 'onnxruntime'. I'm stuck here. Any suggestions?

Ensure that all dependencies for onnxruntime.dll can be found. That includes all the NVIDIA libraries that we can not redistribute and any other things.

jbrownkramer commented 4 years ago

Ensure that all dependencies for onnxruntime.dll can be found. That includes all the NVIDIA libraries that we can not redistribute and any other things.

Point taken. Although in this case, there I am trying to run the openvino backend, so there shouldn't be any NVIDIA drivers. Any other gotcha dependencies I should be aware of?

sfatimar commented 4 years ago

I was able to run squeezenet model with openvino ep. If we set the openvino setup environment , there is no need of adding extra Dlls. Please check for onnxruntime.dll in TestApp\TestApp\bin\Debug\netcoreapp3.1\runtimes\win-x64\native We have created a bug for openvino script.

sfatimar commented 4 years ago

You may also download and install the Intel® Graphics Driver for Windows (26.20) [driver package] before running models on GPU. Please let us know if it resolves your issue