Linux driver i915 SVM support required for Level Zero? If so, upstreaming status?

[simonlui]

Copying mostly over from https://github.com/oneapi-src/level-zero/issues/105 and adding information as I was told to reopen the issue here by @jandres742.

Hello,

I have been trying to get level-zero to work on kernels in a non-supported Linux distribution (Fedora), which I know Intel currently doesn't support, but it confused me why things wouldn't work correctly even with the latest kernel compiled from drm-intel. level-zero recently was added and along with the compute-runtime built from the Fedora packages, I installed both packages and it seems like everything should be in place for things to work but when opening Blender, the oneAPI option doesn't enable for me nor does any sort of other test applications with oneAPI I tinkered with. From looking around, running sycl-ls from the OneAPI packages I installed should list my Intel Arc A770 as a level zero device but I am just plain missing the level zero device output and only see that I have OpenCL device support.

❯ /opt/intel/oneapi/compiler/2023.0.0/linux/bin/sycl-ls
[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2022.15.12.0.01_081451]
[opencl:cpu:1] Intel(R) OpenCL, AMD Ryzen 9 5950X 16-Core Processor             3.0 [2022.15.12.0.01_081451]
[opencl:gpu:2] Intel(R) OpenCL HD Graphics, Intel(R) Graphics [0x56a0] 3.0 [22.43.24558]

There also doesn't seem to be much clarity why it isn't working when all the software packages are there, either in Blender or otherwise. I also went through and reinstalled a custom built version of level-zero I built from source and that didn't work either. So I decided to look at the key differences of what is inside the i915 DKMS module at https://github.com/intel-gpu/intel-gpu-i915-backports and a vanilla kernel driver i915 Kconfig. I found the following big config difference.

config DRM_I915_SVM
...
      Choose this option if you want Shared Virtual Memory (SVM)
      support in i915. With SVM support, one can share the virtual
      address space between a process and the GPU. SVM is supported
      on both integrated and discrete Intel GPUs.

Looking online according to Phoronix (here and here), it seems like it is essential in getting oneAPI and Level Zero to work correctly but there doesn't seem to be any signs of it actually being upstreamed into mainline Linux. This post from the freedesktop drm mailing list seems like the last time it was referenced in the mailing lists still as a RFC in late 2019. So my questions:

1.) Is Intel GPU SVM support a requirement in order to run Level Zero? 2.) If so, is there a path in getting Intel GPU SVM support upstreamed properly into Linux proper?

[frantisekz]

I might by off completely, but, the thing is I haven't enable level-zero support compute-runtime side yet. If you're proficient with rpm build, can you try to apply the following patch on top of intel-compute-runtime and update + install new intel-level-zero package?

diff --git a/intel-compute-runtime.spec b/intel-compute-runtime.spec
index 30a16a7..03e67f2 100644
--- a/intel-compute-runtime.spec
+++ b/intel-compute-runtime.spec
@@ -1,6 +1,6 @@
 %global neo_major 22
-%global neo_minor 43
-%global neo_build 24558
+%global neo_minor 49
+%global neo_build 25018.21

 Name: intel-compute-runtime
 Version: %{neo_major}.%{neo_minor}.%{neo_build}
@@ -29,14 +29,15 @@ BuildRequires: ninja-build
 BuildRequires: libglvnd-devel
 BuildRequires: ocl-icd-devel
 BuildRequires: opencl-headers
-# level-zero to be added later on
+BuildRequires: oneapi-level-zero-devel

 # This doesn't get added automatically, so specify it explicitly
 Requires: intel-igc

-# Let compute-runtime be a meta package for intel-ocloc and intel-opencl
+# Let compute-runtime be a meta package for intel-ocloc, intel-opencl and intel-level-zero
 Requires: intel-ocloc = %{version}-%{release}
 Requires: intel-opencl = %{version}-%{release}
+Requires: intel-level-zero = %{version}-%{release}

 # prelim/drm
 Provides: bundled(drm-uapi-helper)
@@ -76,6 +77,19 @@ on Intel GPUs which basically defines and implements the OpenCL host functions
 required to initialize the device, create the command queues, the kernels and
 the programs and run them on the GPU.

+%package -n    intel-level-zero
+Summary:       oneAPI L0 support implementation for Intel GPUs
+Requires:      intel-igc-libs%{?_isa}
+Requires:      intel-gmmlib%{?_isa}
+# In some references, the package is named intel-level-zero-gpu, so provide that for convenience too
+Provides:      intel-level-zero-gpu%{?_isa}
+
+%description -n intel-level-zero
+Implementation for the Intel GPUs of the oneAPI L0 specification -  which provides direct-to-metal
+interfaces to offload accelerator devices. Its programming interface can be tailored to any device
+needs and can be adapted to support broader set of languages features such as function pointers,
+virtual functions, unified memory, and I/O capabilities..
+
 %prep
 %autosetup -p1 -n compute-runtime-%{version}

@@ -100,7 +114,6 @@ rm -rv third_party/sse2neon
     -DNEO_OCL_VERSION_MAJOR=%{neo_major} \
     -DNEO_OCL_VERSION_MINOR=%{neo_minor} \
     -DNEO_VERSION_BUILD=%{neo_build} \
-    -DSUPPORT_DG1=ON \
     -DSKIP_UNIT_TESTS=1 \
     -DNEO_DISABLE_LD_GOLD=1 \
     -DKHRONOS_GL_HEADERS_DIR="/usr/include/GL/" \
@@ -118,10 +131,12 @@ rm -rv third_party/sse2neon
 %files -n intel-opencl
 %license LICENSE.md
 %{_libdir}/intel-opencl/libigdrcl.so
-# Uncomment once we get level-zero
-#%%{_libdir}/libze_intel_gpu.so*
 %{_sysconfdir}/OpenCL/vendors/intel.icd

+%files -n intel-level-zero
+%license LICENSE.md
+%{_libdir}/libze_intel_gpu.so.*
+
 %files -n intel-ocloc
 %license LICENSE.md
 %{_bindir}/ocloc 

As for the reasons I haven't rebuilt it yet, I was planning to bump the igc/cleanup some patches in one step, but that's blocked due to https://github.com/intel/intel-graphics-compiler/issues/273 currently.

[simonlui]

@frantisekz I am proficient, so I went ahead and did what you asked. Your patch didn't apply entirely cleanly on a clean git checkout of the intel-compute package rawhide branch, but it was just the last chunk that had a conflict so I manually edited the lines in and built and installed the intel-compute packages. What you said seems to have been the issue in Fedora that was blocking oneAPI from showing up and working. It's now showing the level zero GPU line in sycl-ls for me.

❯ /opt/intel/oneapi/compiler/2023.0.0/linux/bin/sycl-ls
[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2022.15.12.0.01_081451]
[opencl:cpu:1] Intel(R) OpenCL, AMD Ryzen 9 5950X 16-Core Processor             3.0 [2022.15.12.0.01_081451]
[opencl:gpu:2] Intel(R) OpenCL HD Graphics, Intel(R) Graphics [0x56a0] 3.0 [22.49.25018.21]
[ext_oneapi_level_zero:gpu:0] Intel(R) Level-Zero, Intel(R) Graphics [0x56a0] 1.3 [1.3.25018]

And Blender recognizes the GPU now.

I did some benchmark runs on Blender afterwards and it seems slow with scores under 1k, but I guess that is somewhat outside of the scope of this issue. Seems great that it was all that it took to get it working and I guess Fedora users will need to wait for everything to trickle down but thank you for your work and maintaining the package for us users.

My hypothesis was wrong then. But I am leaving the issue open for now because it does still seem strange still that these SVM series of patches hasn't gotten into mainline Linux. I'm glad at least it's not essential for getting things to work but it's unclear if it is needed in any significant way for speed or etc, which may explain the above slow Blender test scores.

[eero-t]

I might by off completely, but, the thing is I haven't enable level-zero support compute-runtime side yet. If you're proficient with rpm build, can you try to apply the following patch on top of intel-compute-runtime and update + install new intel-level-zero package? ... As for the reasons I haven't rebuilt it yet, I was planning to bump the igc/cleanup some patches in one step, but that's blocked due to intel/intel-graphics-compiler#273 currently.

FYI: compute runtime package in Debian already enabled Level-Zero support, see discussion in: https://github.com/oneapi-src/level-zero/issues/73

[novemberist]

As for the reasons I haven't rebuilt it yet, I was planning to bump the igc/cleanup some patches in one step, but that's blocked due to intel/intel-graphics-compiler#273 currently.

@frantisekz

Since the issue seems to be fixed now, is there still any good reason to hold back on finally enabling level-zero-support for the Fedora package?

[frantisekz]

As for the reasons I haven't rebuilt it yet, I was planning to bump the igc/cleanup some patches in one step, but that's blocked due to intel/intel-graphics-compiler#273 currently.

@frantisekz

Since the issue seems to be fixed now, is there still any good reason to hold back on finally enabling level-zero-support for the Fedora package?

I've enabled it yesterday in f38/f39: https://src.fedoraproject.org/rpms/intel-compute-runtime/c/4b75ca7295a7debf4f37848e38013ca82b482928?branch=rawhide

You can install intel-compute-runtime (which is meta package that pulls CL and L0) or intel-level-zero there. (Disclaimer: I didn't test the L0 sanity).

I'd like to make it happen in f37 too, however gmmlib'd have to be bumped and it might be unnecessary hassle with f38 coming soon.

[novemberist]

Thanks for heads up and your work on this. I think I will stick to running the oneapi stack through an Arch container for the time being. I guess most packages that make use of oneapi, like blender, would not be recompiled with oneapi support before F38 arrives anyway. And it seems kernel 6.2 is somewhat required for a smooth experience.

[eero-t]

I did some benchmark runs on Blender afterwards and it seems slow with scores under 1k, but I guess that is somewhat outside of the scope of this issue. Seems great that it was all that it took to get it working and I guess Fedora users will need to wait for everything to trickle down but thank you for your work and maintaining the package for us users.

My hypothesis was wrong then. But I am leaving the issue open for now because it does still seem strange still that these SVM series of patches hasn't gotten into mainline Linux. I'm glad at least it's not essential for getting things to work but it's unclear if it is needed in any significant way for speed or etc, which may explain the above slow Blender test scores.

@simonlui Maybe you could check with intel_gpu_top tool [1] whether Blender is keeping GPU busy (freqs up and utilization near 100%), or is its performance bound by something else than GPU side work?

[1] Fedora: igt-gpu-tools package, Debian/Ubuntu: intel-gpu-tools package

[simonlui]

Sorry, I came across this cleaning some other issues I had lying around and I should've closed this a while back but yeah, with the updates made to Blender and the various packages with Level Zero and etc, Blender was able to get to the speeds expected in the benchmark.

Edit: I should mention that i915 SVM support isn't needed and while it would be welcomed, hasn't evidently made its way into mainline so I am hoping that gets rectified at some point.

[eero-t]

Edit: I should mention that i915 SVM support isn't needed and while it would be welcomed, hasn't evidently made its way into mainline so I am hoping that gets rectified at some point.

I assume this flag is for SVM support, and it depends on GTT size: https://github.com/intel/compute-runtime/blob/master/shared/source/os_interface/linux/product_helper_drm.cpp#L131