Closed dmenig closed 3 years ago
To give another use case, maybe easier to reproduce : When I run this
docker run -it -u 0 --device /dev/dri:/dev/dri --device-cgroup-rule='c 189:* rmw' -v /dev/bus/usb:/dev/bus/usb --rm openvino/ubuntu20_dev:latest
/bin/bash -c "apt update && apt install sudo && deployment_tools/demo/demo_security_barrier_camera.sh -d GPU -sample-options -no_show"
It eventually hangs here :
[ 92%] Building CXX object security_barrier_camera_demo/cpp/CMakeFiles/security_barrier_camera_demo.dir/main.cpp.o
[100%] Linking CXX executable ../../intel64/Release/security_barrier_camera_demo
[100%] Built target security_barrier_camera_demo
###################################################
Run Inference Engine security_barrier_camera demo
Run ./security_barrier_camera_demo -d GPU -d_va GPU -d_lpr GPU -i /opt/intel/openvino_2021.3.394/deployment_tools/demo/car_1.bmp -m /root/openvino_models/ir/intel/vehicle-license-plate-detection-barrier-0106/FP16/vehicle-license-plate-detection-barrier-0106.xml -m_lpr /root/openvino_models/ir/intel/license-plate-recognition-barrier-0001/FP16/license-plate-recognition-barrier-0001.xml -m_va /root/openvino_models/ir/intel/vehicle-attributes-recognition-barrier-0039/FP16/vehicle-attributes-recognition-barrier-0039.xml -no_show
[ INFO ] InferenceEngine: API version ......... 2.1
Build ........... 2021.3.0-2787-60059f2c755-releases/2021/3
[ INFO ] Files were added: 1
[ INFO ] /opt/intel/openvino_2021.3.394/deployment_tools/demo/car_1.bmp
[ INFO ] Loading device GPU
[ INFO ] GPU
clDNNPlugin version ......... 2.1
Build ........... 2021.3.0-2787-60059f2c755-releases/2021/3
[ INFO ] Loading detection model to the GPU plugin
*** hangs ***
When it hangs, one cpu core is constantly at 100% and a process named "tracker-miner-fs" seems to be responsible.
Your host system is Ubuntu 20, please follow the Configuration Guide for the Intel® Graphics Compute Runtime for OpenCL™ on Ubuntu* 20.04..
Intel® Graphics Compute Runtime for OpenCL™ driver components are required to use a GPU plugin and write custom layers for Intel® Integrated Graphics. The driver is installed in the OpenVINO™ Docker image but you need to activate it in the container for a non-root user if you have Ubuntu 20.04 on your host. To access GPU capabilities, you need to have correct permissions on the host and Docker container.
Are you sure ? The code snippets I used do work on Ubuntu 20 for intel GPU coming with i3-7200U.
Anyways I have tried the first method on the page you linked :
docker run -it --rm --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render*) openvino/ubuntu20_dev:latest
It doesn't work : it prints this :
groups: cannot find name for group ID 109
[setupvars.sh] OpenVINO environment initialized
openvino@1816ceaf7710:/opt/intel/openvino_2021.3.394$
And then same hanging problem on the scripts mentionned above.
Using the second method at least works but doesn't solve the hanging problem.
veesion@slartibartfast:~$ docker run -it --rm --user root --device /dev/dri --name openvino openvino/ubuntu20_dev:latest
[setupvars.sh] OpenVINO environment initialized
root@c8786dcda200:/opt/intel/openvino_2021.3.394# ls -l /dev/dri/
total 0
crw-rw---- 1 root video 226, 0 Mar 30 11:33 card0
crw-rw---- 1 root 109 226, 128 Mar 30 11:33 renderD128
root@c8786dcda200:/opt/intel/openvino_2021.3.394# addgroup --gid 109 render
addgroup: The group `render' already exists.
root@c8786dcda200:/opt/intel/openvino_2021.3.394# delgroup render Removing group `render' ...
Done.
root@c8786dcda200:/opt/intel/openvino_2021.3.394# addgroup --gid 109 render
Adding group `render' (GID 109) ...
Done.
root@c8786dcda200:/opt/intel/openvino_2021.3.394# ls -l /dev/dri/
total 0
crw-rw---- 1 root video 226, 0 Mar 30 11:33 card0
crw-rw---- 1 root render 226, 128 Mar 30 11:33 renderD128
root@c8786dcda200:/opt/intel/openvino_2021.3.394# usermod -a -G render openvino
root@c8786dcda200:/opt/intel/openvino_2021.3.394# id openvino
uid=1000(openvino) gid=1000(openvino) groups=1000(openvino),44(video),100(users),109(render)
root@c8786dcda200:/opt/intel/openvino_2021.3.394# su openvino
[setupvars.sh] OpenVINO environment initialized
And then same hanging problem on the scripts mentionned above.
Have you test the demo on OpenVINO with iGPU on regular OpenVINO installation or test it without using Docker?
Have you test the demo on OpenVINO with iGPU on regular OpenVINO installation or test it without using Docker?
I only tested with docker.
I tried installing openvino myself after your message following these steps https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_apt.html Then the "Steps for Intel® Processor Graphics (GPU)" in here https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html
GPU is properly listed in available devices aftewards. But I still have the same hang problem.
I also tried building from source with this cmake command
cmake -DCMAKE_BUILD_TYPE=Release -DENABLE_MKL_DNN=OFF -DENABLE_CLDNN=ON -DENABLE_NGRAPH=ON \
-DENABLE_OPENCV=OFF -DENABLE_PYTHON=ON -DPYTHON_EXECUTABLE=/usr/bin/python3 \
-DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.8.so -DPYTHON_INCLUDE_DIR=/usr/include/python3.8 ..
It succeeds and shows GPU in available devices in python, but still the same hanging problem.
Hi @hyperfraise
Which GPU driver version are you on? You can check using clinfo
.
Using your provided instructions, I was able to generate the onnx model, convert to IR and run on GPU device (9th gen). It may be specific to 11th gen, could you try to install this version 20.41.18123 and see if it works?
Regards, Jesus
Hi. This is the initial output of clinfo :
Number of platforms 1
Platform Name Intel(R) OpenCL HD Graphics
Platform Vendor Intel(R) Corporation
Platform Version OpenCL 2.1
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_depth_images cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_icd cl_khr_image2d_from_buffer cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_intel_subgroups cl_intel_required_subgroup_size cl_intel_subgroups_short cl_khr_spir cl_intel_accelerator cl_intel_media_block_io cl_intel_driver_diagnostics cl_khr_priority_hints cl_khr_throttle_hints cl_khr_create_command_queue cl_khr_subgroups cl_khr_il_program cl_intel_spirv_device_side_avc_motion_estimation cl_intel_spirv_media_block_io cl_intel_spirv_subgroups cl_khr_spirv_no_integer_wrap_decoration cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_intel_unified_shared_memory_preview cl_intel_planar_yuv cl_intel_packed_yuv cl_intel_va_api_media_sharing
Platform Host timer resolution 1ns
Platform Extensions function suffix INTEL
Platform Name Intel(R) OpenCL HD Graphics
Number of devices 1
Device Name Intel(R) Gen12LP HD Graphics NEO
Device Vendor Intel(R) Corporation
Device Vendor ID 0x8086
Device Version OpenCL 2.1 NEO
Driver Version 19.41.14441
Device OpenCL C Version OpenCL C 2.0
Device Type GPU
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 48
Max clock frequency 1250MHz
Device Partition (core)
Max number of sub-devices 0
Supported partition types None
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 512x512x512
Max work group size 512
Preferred work group size multiple 32
Max sub-groups per work group 64
Sub-group sizes (Intel) 8, 16, 32
Preferred / native vector sizes
char 16 / 16
short 8 / 8
int 4 / 4
long 1 / 1
half 8 / 8 (cl_khr_fp16)
float 1 / 1
double 1 / 1 (n/a)
Half-precision Floating-point support (cl_khr_fp16)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations No
Double-precision Floating-point support (n/a)
Address bits 64, Little-Endian
Global memory size 19946344448 (18.58GiB)
Error Correction support No
Max memory allocation 4294959104 (4GiB)
Unified memory for Host and Device Yes
Shared Virtual Memory (SVM) capabilities (core)
Coarse-grained buffer sharing Yes
Fine-grained buffer sharing No
Fine-grained system sharing No
Atomics No
Minimum alignment for any data type 128 bytes
Alignment of base address 1024 bits (128 bytes)
Preferred alignment for atomics
SVM 64 bytes
Global 64 bytes
Local 64 bytes
Max size for global variable 65536 (64KiB)
Preferred total size of global vars 4294959104 (4GiB)
Global Memory cache type Read/Write
Global Memory cache size 524288 (512KiB)
Global Memory cache line size 64 bytes
Image support Yes
Max number of samplers per kernel 16
Max size for 1D images from buffer 268434944 pixels
Max 1D or 2D image array size 2048 images
Base address alignment for 2D image buffers 4 bytes
Pitch alignment for 2D image buffers 4 pixels
Max 2D image size 16384x16384 pixels
Max planar YUV image size 16384x16352 pixels
Max 3D image size 16384x16384x2048 pixels
Max number of read image args 128
Max number of write image args 128
Max number of read/write image args 128
Max number of pipe args 16
Max active pipe reservations 1
Max pipe packet size 1024
Local memory type Local
Local memory size 65536 (64KiB)
Max number of constant args 8
Max constant buffer size 4294959104 (4GiB)
Max size of kernel argument 1024
Queue properties (on host)
Out-of-order execution Yes
Profiling Yes
Queue properties (on device)
Out-of-order execution Yes
Profiling Yes
Preferred size 131072 (128KiB)
Max size 67108864 (64MiB)
Max queues on device 1
Max events on device 1024
Prefer user sync for interop Yes
Profiling timer resolution 52ns
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Sub-group independent forward progress Yes
IL version SPIR-V_1.2
SPIR versions 1.2
printf() buffer size 4194304 (4MiB)
Built-in kernels (n/a)
Device Extensions cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_depth_images cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_icd cl_khr_image2d_from_buffer cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_intel_subgroups cl_intel_required_subgroup_size cl_intel_subgroups_short cl_khr_spir cl_intel_accelerator cl_intel_media_block_io cl_intel_driver_diagnostics cl_khr_priority_hints cl_khr_throttle_hints cl_khr_create_command_queue cl_khr_subgroups cl_khr_il_program cl_intel_spirv_device_side_avc_motion_estimation cl_intel_spirv_media_block_io cl_intel_spirv_subgroups cl_khr_spirv_no_integer_wrap_decoration cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_intel_unified_shared_memory_preview cl_intel_planar_yuv cl_intel_packed_yuv cl_intel_va_api_media_sharing
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) Intel(R) OpenCL HD Graphics
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [INTEL]
clCreateContext(NULL, ...) [default] Success [INTEL]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) Success (1)
Platform Name Intel(R) OpenCL HD Graphics
Device Name Intel(R) Gen12LP HD Graphics NEO
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) Success (1)
Platform Name Intel(R) OpenCL HD Graphics
Device Name Intel(R) Gen12LP HD Graphics NEO
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) Success (1)
Platform Name Intel(R) OpenCL HD Graphics
Device Name Intel(R) Gen12LP HD Graphics NEO
ICD loader properties
ICD loader Name OpenCL ICD Loader
ICD loader Vendor OCL Icd free software
ICD loader Version 2.2.11
ICD loader Profile OpenCL 2.1
After doing the steps over on your link, this is what clinfo prints :
Number of platforms 1
Platform Name Intel(R) OpenCL HD Graphics
Platform Vendor Intel(R) Corporation
Platform Version OpenCL 3.0
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_icd cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_intel_subgroups cl_intel_required_subgroup_size cl_intel_subgroups_short cl_khr_spir cl_intel_accelerator cl_intel_driver_diagnostics cl_khr_priority_hints cl_khr_throttle_hints cl_khr_create_command_queue cl_intel_subgroups_char cl_intel_subgroups_long cl_khr_il_program cl_intel_mem_force_host_memory cl_khr_subgroup_extended_types cl_khr_subgroup_non_uniform_vote cl_khr_subgroup_ballot cl_khr_subgroup_non_uniform_arithmetic cl_khr_subgroup_shuffle cl_khr_subgroup_shuffle_relative cl_khr_subgroup_clustered_reduce cl_intel_spirv_media_block_io cl_intel_spirv_subgroups cl_khr_spirv_no_integer_wrap_decoration cl_intel_unified_shared_memory_preview cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_intel_planar_yuv cl_intel_packed_yuv cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_image2d_from_buffer cl_khr_depth_images cl_intel_media_block_io cl_khr_3d_image_writes cl_intel_va_api_media_sharing cl_intel_subgroup_local_block_io
Platform Host timer resolution 1ns
Platform Extensions function suffix INTEL
Platform Name Intel(R) OpenCL HD Graphics
Number of devices 1
Device Name Intel(R) Gen12LP HD Graphics NEO
Device Vendor Intel(R) Corporation
Device Vendor ID 0x8086
Device Version OpenCL 3.0 NEO
Driver Version 20.41.18123
Device OpenCL C Version OpenCL C 1.2
Device Type GPU
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 48
Max clock frequency 1250MHz
Device Partition (core)
Max number of sub-devices 0
Supported partition types None
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 512x512x512
Max work group size 512
Preferred work group size multiple 64
Max sub-groups per work group 64
Sub-group sizes (Intel) 8, 16, 32
Preferred / native vector sizes
char 16 / 16
short 8 / 8
int 4 / 4
long 1 / 1
half 8 / 8 (cl_khr_fp16)
float 1 / 1
double 1 / 1 (n/a)
Half-precision Floating-point support (cl_khr_fp16)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations No
Double-precision Floating-point support (n/a)
Address bits 64, Little-Endian
Global memory size 19946344448 (18.58GiB)
Error Correction support No
Max memory allocation 4294959104 (4GiB)
Unified memory for Host and Device Yes
Shared Virtual Memory (SVM) capabilities (core)
Coarse-grained buffer sharing Yes
Fine-grained buffer sharing No
Fine-grained system sharing No
Atomics No
Minimum alignment for any data type 128 bytes
Alignment of base address 1024 bits (128 bytes)
Preferred alignment for atomics
SVM 64 bytes
Global 64 bytes
Local 64 bytes
Max size for global variable 65536 (64KiB)
Preferred total size of global vars 4294959104 (4GiB)
Global Memory cache type Read/Write
Global Memory cache size 524288 (512KiB)
Global Memory cache line size 64 bytes
Image support Yes
Max number of samplers per kernel 16
Max size for 1D images from buffer 268434944 pixels
Max 1D or 2D image array size 2048 images
Base address alignment for 2D image buffers 4 bytes
Pitch alignment for 2D image buffers 4 pixels
Max 2D image size 16384x16384 pixels
Max planar YUV image size 16384x16352 pixels
Max 3D image size 2048x2048x2048 pixels
Max number of read image args 128
Max number of write image args 128
Max number of read/write image args 128
Max number of pipe args 0
Max active pipe reservations 0
Max pipe packet size 0
Local memory type Local
Local memory size 65536 (64KiB)
Max number of constant args 8
Max constant buffer size 4294959104 (4GiB)
Max size of kernel argument 2048 (2KiB)
Queue properties (on host)
Out-of-order execution Yes
Profiling Yes
Queue properties (on device)
Out-of-order execution No
Profiling No
Preferred size 0
Max size 0
Max queues on device 0
Max events on device 0
Prefer user sync for interop Yes
Profiling timer resolution 52ns
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Sub-group independent forward progress No
IL version SPIR-V_1.2
SPIR versions 1.2
printf() buffer size 4194304 (4MiB)
Built-in kernels (n/a)
Device Extensions cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_icd cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_intel_subgroups cl_intel_required_subgroup_size cl_intel_subgroups_short cl_khr_spir cl_intel_accelerator cl_intel_driver_diagnostics cl_khr_priority_hints cl_khr_throttle_hints cl_khr_create_command_queue cl_intel_subgroups_char cl_intel_subgroups_long cl_khr_il_program cl_intel_mem_force_host_memory cl_khr_subgroup_extended_types cl_khr_subgroup_non_uniform_vote cl_khr_subgroup_ballot cl_khr_subgroup_non_uniform_arithmetic cl_khr_subgroup_shuffle cl_khr_subgroup_shuffle_relative cl_khr_subgroup_clustered_reduce cl_intel_spirv_media_block_io cl_intel_spirv_subgroups cl_khr_spirv_no_integer_wrap_decoration cl_intel_unified_shared_memory_preview cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_intel_planar_yuv cl_intel_packed_yuv cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_image2d_from_buffer cl_khr_depth_images cl_intel_media_block_io cl_khr_3d_image_writes cl_intel_va_api_media_sharing cl_intel_subgroup_local_block_io
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) Intel(R) OpenCL HD Graphics
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [INTEL]
clCreateContext(NULL, ...) [default] Success [INTEL]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) Success (1)
Platform Name Intel(R) OpenCL HD Graphics
Device Name Intel(R) Gen12LP HD Graphics NEO
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) Success (1)
Platform Name Intel(R) OpenCL HD Graphics
Device Name Intel(R) Gen12LP HD Graphics NEO
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) Success (1)
Platform Name Intel(R) OpenCL HD Graphics
Device Name Intel(R) Gen12LP HD Graphics NEO
ICD loader properties
ICD loader Name OpenCL ICD Loader
ICD loader Vendor OCL Icd free software
ICD loader Version 2.2.11
ICD loader Profile OpenCL 2.1
NOTE: your OpenCL library only supports OpenCL 2.1,
but some installed platforms support OpenCL 3.0.
Programs using 3.0 features may crash
or behave unexpectedly
I conclude that I updated the driver successfully. This solves the hanging problem as well (although I'm noting that the loading of the model in iGPU memory is very slow : ~2m30s).
If I may say, I believe that this incompatibility should be listed somewhere ? Or at least there should be some indication anywhere on Openvino's docs that this driver should be the most recent one, idk.
Thank you very much !
@hyperfraise
This solves the hanging problem as well (although I'm noting that the loading of the model in iGPU memory is very slow : ~2m30s).
Was the loading time similar on the previous driver? Take a look at this support article on how to load model from cache.
Regards, Jesus
As noted above, there couldn'tbe any loading on the previous driver : just hanging.
@hyperfraise oh yes.... my mistake. Loading on GPU will take a bit longer as the kernel compiles, the article mentioned in my previous response should help when reloading the model after the first time.
System information (version)
Detailed description
Openvino code hangs on this hardware using the iGPU, but it works with the CPU, on a very common model.
Steps to reproduce
First run this to export the onnx model.
Then I enter the official container for openvino :
docker run --name openvino -it --device /dev/dri:/dev/dri --device-cgroup-rule='c 189:* rmw' -v /dev/bus/usb:/dev/bus/usb --rm openvino/ubuntu20_dev:latest
Then I run the conversion script :
Then simply loading the model hangs :
I verified that the same code works with CPU, or that it works with another iGPU (i3-7200U).
Although such a script still shows the GPU is available :