Unable to get working OpenVINO/ONNX GPU accelleration on Ubuntu 24.04, Intel i7 1355U

[khhsenti]

OpenVINO Version

onnxruntime-openvino 1.17.1

Operating System

Other (Please specify in description)

Hardware Architecture

x86 (64 bits)

Target Platform

DT Research tablet DT302-RP with Intel i7 1355U , running Ubuntu 24.04 LTS

Build issue description

We are trying to evaluate the possible performance improvement when running AI inference through the OpenVINO framework. Our AI models are ONNX based, and as such the integration between OpenVINO and ONNX need to work.

We have tried using the provided demonstration Docker container, but it is unable to find the GPU.

Installing the OpenVINO framework on the host OS directly, does provide the GPU as an option, but the Python version on Ubuntu 24.04 is too new that there is a PIP package available for the OpenVINO ONNX integration. We have then continued to build the ONNX integration manually, which worked, but the OpenVINO execution provider does not load.

Build script or step-by-step to reproduce

Run the provided docker container:

sentispec-admin@sentispec-DT302RP-dsv-1:~$ docker run -it --rm --device-cgroup-rule='c 189:* rmw' -v /dev/bus/usb:/dev/bus/usb --device /dev/dri:/dev/dri openvino/onnxruntime_ep_ubuntu20:latest
onnxruntimedev@f997be011eee:~$ python3 -c "from openvino import Core; print(Core().available_devices)"
['CPU', 'GNA']
onnxruntimedev@f997be011eee:~$ pip list
Package              Version
-------------------- --------------------
coloredlogs          15.0.1
dbus-python          1.2.16
distro-info          0.23+ubuntu1.1
flatbuffers          23.5.26
humanfriendly        10.0
mpmath               1.3.0
numpy                1.24.4
onnxruntime-openvino 1.17.1
packaging            23.2
pip                  23.3.2
protobuf             4.25.3
PyGObject            3.36.0
python-apt           2.0.1+ubuntu0.20.4.1
setuptools           45.2.0
sympy                1.12
unattended-upgrades  0.1
wheel                0.34.2

[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: python3.8 -m pip install --upgrade pip
onnxruntimedev@f997be011eee:~$

Install OpenVINO manually on host OS:

Follow: https://docs.openvino.ai/2022.3/openvino_docs_install_guides_configurations_for_intel_gpu.html

pip install openvino newest version

root@sentispec-DT302RP-dsv-1:~# source openvino_env/bin/activate (openvino_env) root@sentispec-DT302RP-dsv-1:~# python3 -c "from openvino import Core; print(Core().available_devices)" ['CPU', 'GPU'] (openvino_env) root@sentispec-DT302RP-dsv-1:~#

Build ONNX with OpenVino wheel to manual installation:

https://onnxruntime.ai/docs/build/eps.html#linux-4

pip install /home/sentispec-admin/onnxruntime_openvino-1.19.0-cp312-cp312-linux_x86_64.whl

Try to use it:

(openvino_env) root@sentispec-DT302RP-dsv-1:~# python3
Python 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnxruntime as ort
>>> session = ort.InferenceSession("/path/to/model", providers=["OpenVINOExecutionProvider"])
/root/openvino_env/lib/python3.12/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:69: UserWarning: Specified provider 'OpenVINOExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'
  warnings.warn(

Happy to provide any details, decided to keep it as short as possible for now. I am aware ubuntu 24.04 is not currently officially supported but was thinking that it is planned e.g. also regarding this: https://github.com/openvinotoolkit/openvino/pull/23905

Relevant log output

No response

Issue submission checklist

• [X] I'm reporting an issue. It's not a question.
• [X] I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.

[ilya-lavrenov]

@preetha-intel

[jatinwadhwa921]

Hi @ilya-lavrenov ,

Installation of ONNX Runtime with OpenVINO

For installing ONNX Runtime, rather than using pip install openvino, please follow these steps:

1. Download OpenVINO from the following archive: OpenVINO 2024.1 Linux.
2. Extract the archive and run the setupvars.sh script located in the extracted OpenVINO directory to set up the environment variables.
3. Build ONNX Runtime after setting up OpenVINO; this should resolve any installation issues.

Running Docker Container

Please use the following command to run the Docker container:

docker run -it --rm --device-cgroup-rule='c 189:* rmw' -v /dev/bus/usb:/dev/bus/usb --device /dev/dri:/dev/dri --group-add=$(stat -c "%g" /dev/dri/render*) -v docker_val/:/docker_val <image_name:tag>

This command ensures that the container has the same group permissions as the host for the Direct Rendering Manager (DRM) devices, allowing you to perform inference using GPU devices.

We have tested this setup on Ubuntu 20.04, and i believe it should work on Ubuntu 24.04 as well.

[khhsenti]

tried running the more elaborate docker command but does not seem to work:

Status: Downloaded newer image for openvino/onnxruntime_ep_ubuntu20:latest
groups: cannot find name for group ID 992
onnxruntimedev@a358b8916a2d:~$ python3 -c "from openvino import Core; print(Core().available_devices)"
['CPU', 'GNA']
onnxruntimedev@a358b8916a2d:~$

[khhsenti]

tried to add the group in after the container has been created:

docker exec --user=root -it magical_chaplygin groupadd -g 992 render

But still does not work:

test@test-DT302RP:~/openvino$ docker run -it --rm --device-cgroup-rule='c 189:* rmw' -v /dev/bus/usb:/dev/bus/usb --device /dev/dri:/dev/dri --group-add=$(stat -c "%g" /dev/dri/render*) --user root openvino/onnxruntime_ep_ubuntu20:latest
groups: cannot find name for group ID 992
root@ad4326d20d2c:/home/onnxruntimedev# groups
root render
root@ad4326d20d2c:/home/onnxruntimedev# python3 -c "from openvino import Core; print(Core().available_devices)"
['CPU', 'GNA']
root@ad4326d20d2c:/home/onnxruntimedev# ls -l /dev/dri/renderD128
crw-rw---- 1 root render 226, 128 Jun 10 07:03 /dev/dri/renderD128
root@ad4326d20d2c:/home/onnxruntimedev#

[jatinwadhwa921]

Hi, can you share the current GPU driver version that is installed in your ubuntu 24.04 OS and the link for installing the driver

[ncs1]

I'm having the same issue, GPU (ARC A770) doesn't identify in hello_query_device / clinfo inside docker environment. Running ub24.04 latest kernel 6.9.5 (checked against ubuntu 6.8 generic one too) Checked different openvino versions, and other images (intelanalytics). Running the container with privileged, passing group id render,video doesn't fix it.

[khhsenti]

Hi, can you share the current GPU driver version that is installed in your ubuntu 24.04 OS and the link for installing the driver

Sure. This is "stock Ubuntu 24.04 LTS", have not installed anything additional. So I don't think I have more details than the kernel version and how the driver informs us in dmesg. It did not look like you have to install a out-of-tree driver to make it work? As OpenVino does find the GPU on the host.

sentispec-admin@sentispec-DT302RP-1:~$ uname -a
Linux sentispec-DT302RP-dhl-1 6.8.0-35-generic #35-Ubuntu SMP PREEMPT_DYNAMIC Mon May 20 15:51:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
sentispec-admin@sentispec-DT302RP-dhl-1:~$ sudo dmesg | grep i915
[sudo] password for sentispec-admin:
[    2.127483] i915 0000:00:02.0: [drm] VT-d active for gfx access
[    2.145374] i915 0000:00:02.0: vgaarb: deactivate vga console
[    2.145538] i915 0000:00:02.0: [drm] Using Transparent Hugepages
[    2.146074] i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    2.148779] i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc.bin (v2.20)
[    2.180623] i915 0000:00:02.0: [drm] GT0: GuC firmware i915/adlp_guc_70.bin version 70.20.0
[    2.180631] i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[    2.204640] i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads
[    2.205161] i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[    2.205171] i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[    2.205907] i915 0000:00:02.0: [drm] GT0: GUC: RC enabled
[    2.207523] i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[    3.346322] [drm] Initialized i915 1.6.0 20230929 for 0000:00:02.0 on minor 1
[    3.355474] i915 display info: display version: 13
[    3.355478] i915 display info: cursor_needs_physical: no
[    3.355480] i915 display info: has_cdclk_crawl: yes
[    3.355482] i915 display info: has_cdclk_squash: no
[    3.355484] i915 display info: has_ddi: yes
[    3.355486] i915 display info: has_dp_mst: yes
[    3.355488] i915 display info: has_dsb: yes
[    3.355490] i915 display info: has_fpga_dbg: yes
[    3.355492] i915 display info: has_gmch: no
[    3.355493] i915 display info: has_hotplug: yes
[    3.355495] i915 display info: has_hti: no
[    3.355497] i915 display info: has_ipc: yes
[    3.355498] i915 display info: has_overlay: no
[    3.355500] i915 display info: has_psr: yes
[    3.355502] i915 display info: has_psr_hw_tracking: no
[    3.355503] i915 display info: overlay_needs_physical: no
[    3.355505] i915 display info: supports_tv: no
[    3.355507] i915 display info: has_hdcp: yes
[    3.355508] i915 display info: has_dmc: yes
[    3.355510] i915 display info: has_dsc: yes
[    3.360394] fbcon: i915drmfb (fb0) is primary device
[    3.360405] i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device

[ncs1]

@khhsenti I just setup my own docker from base 24.04 image, added their installs and got gpu acceleration successfully.