ze_init returns ZE_RESULT_ERROR_UNINITIALIZED

[maleadt]

I'm experimenting with oneL0, but on my system the very first call to ze_init returns ZE_RESULT_ERROR_UNINITIALIZED. For example, using https://github.com/alcf-perfengr/alcl:

$ git clone https://github.com/alcf-perfengr/alcl                                                                                                                                                                                                              
$ cd alcl/L0                              

$ make CFLAGS="-I /usr/include/level_zero -lze_loader"                                                                                                                                                                                        $ ./tiny_zeinfo                                                                                                                                                                                                                               [master] 
>>> Initializing L0  Platform and Device...
Non-successful return code 2013265921 (ZE_RESULT_ERROR_UNINITIALIZED) for zeInit.  Exiting.

I'm on Arch Linux, kernel 5.5.13, using intel-compute-runtime 20.12.16259 and level-zero-loader 0.91.10 from the Arch user repositories. clinfo output:

$ docker run -it --device /dev/dri:/dev/dri --rm docker.io/intelopencl/intel-opencl:ubuntu-18.04-ppa clinfo
Number of platforms                               1
  Platform Name                                   Intel(R) OpenCL HD Graphics
  Platform Vendor                                 Intel(R) Corporation
  Platform Version                                OpenCL 2.1 
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_icd cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_intel_subgroups cl_intel_required_subgroup_size cl_intel_subgroups_short cl_khr_spir cl_intel_accelerator cl_intel_driver_diagnostics cl_khr_priority_hints cl_khr_throttle_hints cl_khr_create_command_queue cl_intel_subgroups_char cl_intel_subgroups_long cl_khr_fp64 cl_khr_subgroups cl_khr_il_program cl_intel_spirv_device_side_avc_motion_estimation cl_intel_spirv_media_block_io cl_intel_spirv_subgroups cl_khr_spirv_no_integer_wrap_decoration cl_intel_unified_shared_memory_preview cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_intel_planar_yuv cl_intel_packed_yuv cl_intel_motion_estimation cl_intel_device_side_avc_motion_estimation cl_intel_advanced_motion_estimation cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_image2d_from_buffer cl_khr_depth_images cl_intel_media_block_io cl_khr_3d_image_writes 
  Platform Host timer resolution                  1ns
  Platform Extensions function suffix             INTEL

  Platform Name                                   Intel(R) OpenCL HD Graphics
Number of devices                                 1
  Device Name                                     Intel(R) Gen9 HD Graphics NEO
  Device Vendor                                   Intel(R) Corporation
  Device Vendor ID                                0x8086
  Device Version                                  OpenCL 2.1 NEO 
  Driver Version                                  20.11.16158
  Device OpenCL C Version                         OpenCL C 2.0 
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Max compute units                               24
  Max clock frequency                             1150MHz
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None, None
  Max work item dimensions                        3
  Max work item sizes                             256x256x256
  Max work group size                             256
  Preferred work group size multiple              32
  Max sub-groups per work group                   32
  Sub-group sizes (Intel)                         8, 16, 32
  Preferred / native vector sizes                 
    char                                                16 / 16      
    short                                                8 / 8       
    int                                                  4 / 4       
    long                                                 1 / 1       
    half                                                 8 / 8        (cl_khr_fp16)
    float                                                1 / 1       
    double                                               1 / 1        (cl_khr_fp64)
  Half-precision Floating-point support           (cl_khr_fp16)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Single-precision Floating-point support         (core)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  Yes
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Address bits                                    64, Little-Endian
  Global memory size                              26845274112 (25GiB)
  Error Correction support                        No
  Max memory allocation                           4294959104 (4GiB)
  Unified memory for Host and Device              Yes
  Shared Virtual Memory (SVM) capabilities        (core)
    Coarse-grained buffer sharing                 Yes
    Fine-grained buffer sharing                   No
    Fine-grained system sharing                   No
    Atomics                                       No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       1024 bits (128 bytes)
  Preferred alignment for atomics                 
    SVM                                           64 bytes
    Global                                        64 bytes
    Local                                         64 bytes
  Max size for global variable                    65536 (64KiB)
  Preferred total size of global vars             4294959104 (4GiB)
  Global Memory cache type                        Read/Write
  Global Memory cache size                        524288 (512KiB)
  Global Memory cache line size                   64 bytes
  Image support                                   Yes
    Max number of samplers per kernel             16
    Max size for 1D images from buffer            268434944 pixels
    Max 1D or 2D image array size                 2048 images
    Base address alignment for 2D image buffers   4 bytes
    Pitch alignment for 2D image buffers          4 pixels
    Max 2D image size                             16384x16384 pixels
    Max planar YUV image size                     16384x16352 pixels
    Max 3D image size                             16384x16384x2048 pixels
    Max number of read image args                 128
    Max number of write image args                128
    Max number of read/write image args           128
  Max number of pipe args                         16
  Max active pipe reservations                    1
  Max pipe packet size                            1024
  Local memory type                               Local
  Local memory size                               65536 (64KiB)
  Max number of constant args                     8
  Max constant buffer size                        4294959104 (4GiB)
  Max size of kernel argument                     1024
  Queue properties (on host)                      
    Out-of-order execution                        Yes
    Profiling                                     Yes
  Queue properties (on device)                    
    Out-of-order execution                        Yes
    Profiling                                     Yes
    Preferred size                                131072 (128KiB)
    Max size                                      67108864 (64MiB)
  Max queues on device                            1
  Max events on device                            1024
  Prefer user sync for interop                    Yes
  Profiling timer resolution                      83ns
  Execution capabilities                          
    Run OpenCL kernels                            Yes
    Run native kernels                            No
    Sub-group independent forward progress        Yes
    IL version                                    SPIR-V_1.2 
    SPIR versions                                 1.2 
  printf() buffer size                            4194304 (4MiB)
  Built-in kernels                                block_motion_estimate_intel;block_advanced_motion_estimate_check_intel;block_advanced_motion_estimate_bidirectional_check_intel;
  Motion Estimation accelerator version (Intel)   2
    Device-side AVC Motion Estimation version     1
      Supports texture sampler use                Yes
      Supports preemption                         No
  Device Extensions                               cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_icd cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_intel_subgroups cl_intel_required_subgroup_size cl_intel_subgroups_short cl_khr_spir cl_intel_accelerator cl_intel_driver_diagnostics cl_khr_priority_hints cl_khr_throttle_hints cl_khr_create_command_queue cl_intel_subgroups_char cl_intel_subgroups_long cl_khr_fp64 cl_khr_subgroups cl_khr_il_program cl_intel_spirv_device_side_avc_motion_estimation cl_intel_spirv_media_block_io cl_intel_spirv_subgroups cl_khr_spirv_no_integer_wrap_decoration cl_intel_unified_shared_memory_preview cl_khr_mipmap_image cl_khr_mipmap_image_writes cl_intel_planar_yuv cl_intel_packed_yuv cl_intel_motion_estimation cl_intel_device_side_avc_motion_estimation cl_intel_advanced_motion_estimation cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_image2d_from_buffer cl_khr_depth_images cl_intel_media_block_io cl_khr_3d_image_writes 

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  Intel(R) OpenCL HD Graphics
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [INTEL]
  clCreateContext(NULL, ...) [default]            Success [INTEL]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 Intel(R) OpenCL HD Graphics
    Device Name                                   Intel(R) Gen9 HD Graphics NEO
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 Intel(R) OpenCL HD Graphics
    Device Name                                   Intel(R) Gen9 HD Graphics NEO
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 Intel(R) OpenCL HD Graphics
    Device Name                                   Intel(R) Gen9 HD Graphics NEO

ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.2.11
  ICD loader Profile                              OpenCL 2.1

Permissions on the device nodes are OK too:

crw-rw-rw- 1 root render 226, 128 Mar 30 21:36 /dev/dri/renderD128
crw-rw-rw- 1 root render 226, 129 Mar 30 21:36 /dev/dri/renderD129

Although this is probably a misconfiguration of some sort, what are the guarantees about calling ze_init? Is it possible for it to fail when, e.g., the user doesn't have compatible hardware or a driver?

[rwmcguir]

Is there an intel-level-zero-gpu package installed? This is more likely a bug in not finding the associated runtime. Above you are showing the OpenCL driver, but that is not a driver for Level Zero.

Do you have package similar to this installed: intel-level-zero-gpu_0.8.16259_amd64.deb Since I am not that familiar with ArchLinux it would be worth mentioning if this is installed, where the actual shared library got installed (i.e. show the path) and the SONAME.

If you don't have this package or similar then this is the bug. :-)

[maleadt]

I assumed the compute runtime contained a level zero driver after https://github.com/intel/compute-runtime/commit/27f4bce42f614cb86f437106abc666bc7a3f460c, if not where do I get it from? I'm currently using the Arch packages to do some initial experimentation, but for the Julia oneAPI support I'm working on I plan to redistribute necessary dependencies myself.

[rwmcguir]

That SHA should be fine yes... Question is how and what did you install? The repository is capable of building both Level Zero and OpenCL. https://github.com/intel/compute-runtime/releases/tag/20.12.16259 Please note that OpenCL and Level Zero are different packages.

Can you show me a list of all the level zero packages current installed in the system?

If this was Debian i'd ask for : dpkg --list | grep "level-zero" but I am not sure what ArchLinux uses for package manager, or if you built installed yourself, then what list of binaries did you get and install? Hopefully you have a libze_intel_gpu.so.0.8.16259 someplace in the searchable runtime paths. $ cd / find . | grep libze_intel_gpu

[maleadt]

There isn't a specific level zero package on Arch, just the compute runtime built like this: https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/intel-compute-runtime -- maybe it is incorrectly built, or doesn't install the level zero driver?

I have version 20.12.16259 of that package, and thus the runtime, which should include the hash I linked to as per the Github UI.

EDIT: ah, didn't properly read your comment, no it does not ship libze_intel_gpu. I'll build it myself.

[rwmcguir]

There is definitely one bug that needs fixed here in the loader. It needs a proper error message when not finding a value driver, segfaulting is not a great error message.

[rwmcguir]

Also not sure who maintains the inclusion into the ArchLinux for compute-runtime, but it may be worth pinging them and letting them know that Level Zero uses different standard versioning, i.e. SEMVER vs the date based versioning schema for the OpenCL packages. This may not work so well going forward them using a singular package for both different products.

[maleadt]

Rebuilding the package gave me a libze_intel_gpu, strange how it wasn't included in the Arch package. I filed a bug.

On a related note, why isn't the level zero driver included in the intelopencl/intel-opencl:ubuntu-18.04-ppa docker images?

[maleadt]

On a related note, why isn't the level zero driver included in the intelopencl/intel-opencl:ubuntu-18.04-ppa docker images?

Or, related I guess, why the level-zero package isn't available on the ppa that is used in the docker images: https://launchpad.net/~intel-opencl/+archive/ubuntu/intel-opencl

[rwmcguir]

On a related note, why isn't the level zero driver included in the intelopencl/intel-opencl:ubuntu-18.04-ppa docker images?

Level Zero and OpenCL as a product are not the same. Today only the Intel and OpenCL specific implementations share a chunk of common code for gpu implementation, but their interfaces, specifications, ABI/API's and products that support that are completely different. Level Zero is a vendor agnostic loader. No single vendor should be shipping it inside a specific HW implementation package. This is not to dissimilar to the the loader that OpenCL uses. This package should eventually support any number of vendors and HW implementations. This it should not belong inside any of them or you will have package collisions.

Also Linux standards from major distro's call out that stand alone shared libraries should ALWAYS ship in a single package that is named and versioned per the share library being contained. FYI, This is something that ArchLinux seems to consistently violate and it makes vendor support very very difficult... Having these stand alone shared library packages is very necessary to manage package dependencies at install time, through things like yum, dnf, yast, yast2, dpkg, etc... If you have time here is a great read: https://www.debian.org/doc/debian-policy/ch-sharedlibs.html#run-time-shared-libraries

Without policies like this being followed it becomes nearly impossible to upgrade things. Imagine having to replace your entire /usr/ partition each time glibc patched itself. This is the model we are trying to avoid with good practices.

[maleadt]

I get all that, and I'm not asking anything about packages, but why the example Docker images built on the "Intel(R) Graphics Compute Runtime" PPA don't include a Level Zero driver (not installed, but also not available in the PPA). I had hoped to be able to use that image for CI purposes.

This is something that ArchLinux seems to consistently violate and it makes vendor support very very difficult...

I'm not an Arch developer, just a user... FWIW, for the Julia package I'm developing I was hoping to ship the Level Zero loader and (if applicable) the matching NEO implementation myself, but apparently opencl-clang, as required by the intel-graphics-compiler, isn't cross-compilable, and Julia's infrastructure for building binaries for users is based on cross compilation. So for now I'm using my distro's packages.

[rwmcguir]

So there are a lot of things that likely will come along in the near future. This current release is a early preview, hence the pre-1.0 version schema both for the loader and today for the intel-level-zero-gpu package. If you are putting this into production, do realize things will be changing from an API compatibility perspective fairly rapidly. SEMVER isn't a solid guarantee pre-1.0.

There are some preliminary plans to get some kind of CI online for the public, which could then be borrowed, this is still being worked. Perhaps in the future a docker image will exist for oneAPI and/or Level Zero related content. However, it should be noted here that CI for the Level Zero (loader) will not include vendor specific stacks.

Level Zero is part of the oneAPI industry initiative, today OpenCL is not, so it doesn't make sense from a specification standpoint to merge these goals or products. https://spec.oneapi.com/versions/latest/index.html

but apparently opencl-clang, as required by the intel-graphics-compiler, isn't cross-compilable,

I would encourage filing this issue over in the www.github.com/intel/compute-runtime

Thank you for the feedback. It is good to receive the pain points and how the community is using this.

[maleadt]

However, it should be noted here that CI for the Level Zero (loader) will not include vendor specific stacks.

Sure, I just asked the question here since I had an issue open :-) Thanks for the replies!