Intel graphics acceleration architecturally broken for non-Ubuntu systems

shoffmeister commented 1 year ago

Windows build number:

10.0.22621.0

Your Distribution version:

any non-Debian / non-Ubuntu

Your WSL versions:

WSL version: 1.1.3.0 Kernel version: 5.15.90.1 WSLg version: 1.0.49 MSRDC version: 1.2.3770 Direct3D version: 1.608.2-61064218 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows version: 10.0.22621.1344

Steps to reproduce:

WSLg graphics (and video) acceleration is fundamentally and architecturally broken for Intel GPUs on any distribution that is not part of the Debian or Ubuntu world.

In other words, Intel graphics acceleration will not work for WSLg on affected distributions.

To reproduce on any affected distribution,

pick some hardware which has Intel GPU hardware
recent enough Intel drivers supporting WSLg in general, on Ubuntu
open a WSL2 prompt on the affected distribution (assuming that a full Mesa stack including d3d12 DRI driver is installed)
run MESA_D3D12_DEFAULT_ADAPTER_NAME=intel glxinfo -B | grep -oP '(?<=Device: )(.+)$'

//exp: D3D12 (Intel(R) UHD Graphics) (0xffffffff) //act: llvmpipe (LLVM 15.0.7, 256 bits) (0xffffffff)

Affected are, amongst others,

Fedora Linux (https://apps.microsoft.com/store/detail/fedora-remix-for-wsl, https://github.com/WhitewaterFoundry/Fedora-Remix-for-WSL)
OpenSUSE Tumbleweed (https://github.com/openSUSE/WSL-DistroLauncher, https://apps.microsoft.com/store/detail/opensuse-tumbleweed)
Arch Linux (https://apps.microsoft.com/store/detail/arch-wsl, https://github.com/VsTechDev/Arch-WSL)

but also any other custom-build or custom-loaded distribution which does not originate from the Debian world.

WSL logs:

No response

WSL dumps:

No response

Expected behavior:

run MESA_D3D12_DEFAULT_ADAPTER_NAME=intel glxinfo -B | grep -oP '(?<=Device: )(.+)$'

//exp: D3D12 (Intel(R) UHD Graphics) (0xffffffff)

Actual behavior:

run MESA_D3D12_DEFAULT_ADAPTER_NAME=intel glxinfo -B | grep -oP '(?<=Device: )(.+)$'

//act: llvmpipe (LLVM 15.0.7, 256 bits) (0xffffffff)

shoffmeister commented 1 year ago

Note: The steps above use environment variable MESA_D3D12_DEFAULT_ADAPTER_NAME to explicitly pick the Intel GPU; this will be beneficial for reproducing on systems such as notebooks with a hybrid GPU architecture (Intel iGPU, Nvidia dGPU). I have such an Nvidia "Optimus" system (Dell 7610 with Nvidia RTX 3060)

shoffmeister commented 1 year ago

Note: I reproduced the problem specifically for Fedora Linux through "Fedora Remix for Windows Subsystem for Linux", see https://github.com/WhitewaterFoundry/Fedora-Remix-for-WSL

The key value-add of that specific distribution is that it bundles a complete set of current Mesa DRI drivers, including, specifically, the Mesa d3d12_dri.so driver. Out-of-the box Fedora does ship Mesa, but does not ship the D3D12 driver required to enable WSLg graphics acceleration in general.

Obviously any (other) distribution used in trying to reproduce this breakage needs to contain the Mesa d3d12_dri.so driver.

shoffmeister commented 1 year ago

The root cause of this breakage is, in the end, quite simple:

WSLg GPU drivers are mounted via the WSLg system distribution, see

wsl --debug-shell
find /gpu_drivers -type f | grep -E '\.so\.|.so$'

and the matching content in C:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_fc51d29440c1dd25 (this is my current Intel driver version, FWIW - this location may change over time)

The Intel GPU driver for WSLg is a Linux ELF binary.

From within a specific affected distribution, running readelf --dynamic /usr/lib/wsl/drivers/iigd_dch.inf_amd64_8a4323c80a901a5c/libLLVM-9.so | grep '(NEEDED)' will yield

 0x0000000000000001 (NEEDED)             Shared library: [libedit.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]

The reference to [libedit.so.2] is the problem - not on Ubuntu, but everywhere else:

libedit is https://www.thrysoee.dk/editline/ - and we are now entering the exciting world of packaging and SONAMEs:

Fedora knows this as libedit (dnf info libedit) and installs libedit.so.0 and libedit.so.0.0.69
Ubuntu knows this as libedit2 (apt info libedit2) and installs libedit.so.2 and libedit.so.2.0.70

Because the Intel driver (presumably) was built on Ubuntu, the linked SONAME is libedit.so.2. But Fedora Linux does not know anything about libedit.so.2. As a consequence, the Intel driver stack for WSLg will fail to load on Fedora Linux, and graphics acceleration on WSLg is broken.

The resolution on Fedora Linux is straight-forward: Run sudo ln -s /usr/lib64/libedit.so.0 /usr/lib64/libedit.so.2 and MESA_D3D12_DEFAULT_ADAPTER_NAME=intel glxinfo -B | grep -oP '(?<=Device: )(.+)$' will show the expected D3D12 driver mapping.

shoffmeister commented 1 year ago

Apparently there is a bit of a history with libedit - I found https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1749958.html -, but the gist of the matter appears to be straight-forward:

Debian/Ubuntu-based Linux distributions will ship editline/libedit as "libedit.so.2"
everybody else ... not; they will ship "libedit.so.0" (verified on Arch and found through https://pkgs.org/search/?q=libedit&on=provides)

The architectural problem for, and with, WSLg is that

there is exactly one WSL2 system distribution (wsl --debug-shell) providing the infrastructure for any and all "user" Linux distributions that one might wish to run. This implies that anything the system distribution provisions must be 100% compatible with any attached user distribution; there is no way to inject distro-specific artefacts, as the WSL2 system distribution is totally unaware of where it is mapped into.
the WSL2 system distribution provisions a vendor (Intel) driver package which is tightly coupled with a specific build system (via SONAME); this is similar in effect to the challenges that symbol versioning, for instance on glibc, offers for cross-distribution binary builds and packages. This tight coupling runs into the risk that "the driver" is not universally compatible with all user distros.

I do believe that there is no room for the WSLg graphics acceleration architecture to change, though. The observable decisions made are very reasonable from a layering point of view, and having the vendor package the WSLg Linux acceleration components together with the Windows graphics driver helps avoid total dependency chaos.

Anyway, in this case Intel is a bit between a rock and a hard place:

they need to build their libLLVM somewhere,
they will prefer to not touch the implementation to remove the dependency on editline (it escapes me why libLLVM has a direct dependency on editline, to be honest)
dynamically loading either libedit.so.0 or libedit.so.2 and manually wiring up calls is just awful

elebeaup-re4a commented 1 year ago

@shoffmeister Awesome. I was looking for a long time why Intel graphics acceleration was not working. I have the same issue on NixOs and now thanks to your workaround it works.

ryuheechul commented 1 year ago

@elebeaup-re4a I believe I'm experiencing the same issue. But as I'm quite new to NixOS, I couldn't figure out how to resolve this. Could you share how you applied this workaround on NixOS?

Kangie commented 2 months ago

This is also broken on Gentoo in a way that is unable to be fixed downstream.

Please bundle the lib if you need to, or build it against a sane soname.

https://bugs.gentoo.org/937851

Any indication as to when we can expect a fix for this issue? Do we need to report this to Intel? What is the appropriate mechanism for that?

Kangie commented 2 months ago

(it escapes me why libLLVM has a direct dependency on editline, to be honest)

It appears based on this ticket that it exists as part of the llvm line editing library, and there is a fallback implementation using fgets if this dependency is not available (or not desired).

In Gentoo we even support this as an option when building LLVM.

I don't know who is capable of fixing this issue, but the solution is probably -DLLVM_ENABLE_LIBEDIT=no.

microsoft / wslg