search

nix-community / NixOS-WSL

NixOS on WSL(2) [maintainer=@nzbr]
Apache License 2.0
1.75k stars 115 forks source link

Hardware acceleration does not work without LD_LIBRARY_PATH hacks #454

Open Pleune opened 5 months ago

Pleune commented 5 months ago

Bug description

At this time, the WSL2/WSLg D3D12 hardware acceleration does not work out of the box due to library problems, even though the mesa.drivers package does have D3D12 support enabled by default.

I do not understand exactly what is going on. but there are a few ways to set LD_LIBRARY_PATH as a hack to fix it:

In both examples above, /run/opengl-driver/lib can be swapped with /usr/lib/wsl/lib. I'm not sure what the difference is here, other than that /usr/lib/wsl/lib has many more files.

I believe this could be two problems:

  1. the mesa.drivers files are not able to dlopen the WSLg files, since it has no way to know where they are.
  2. The Intel version of the D3D12 sub-driver-??? seems to need some additional libraries to link to that are not currently available.

To Reproduce

With the following config:


  programs.nix-ld = {
    enable = true;
    # I am using this nix-ld-rs too:
    #   package = nix-ld-rs.packages."${pkgs.system}".nix-ld-rs;
  };
  wsl = {
    enable = true;
    defaultUser = "${username}";
    nativeSystemd = true;
    useWindowsDriver = true;
    startMenuLaunchers = true;
  };

  hardware.opengl = {
    enable = true;
    driSupport = true;

    extraPackages = with pkgs; [
      mesa.drivers
      libvdpau-va-gl
      vaapiVdpau
    ];
  };

Then run any of the following commands:

glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel LD_LIBRARY_PATH=ub:/run/opengl-driver/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia LD_LIBRARY_PATH=/run/opengl-driver/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel LD_LIBRARY_PATH=/usr/lib/wsl/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia LD_LIBRARY_PATH=ub:/usr/lib/wsl/lib glxinfo -B

The ub folder I am using is: ub.tar.gz

Logs

Example of hardware acceleration working:

> LD_LIBRARY_PATH=ub:/usr/lib/wsl/lib glxinfo -B
name of display: :0
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: Some incorrect rendering might occur because the selected Vulkan device (Microsoft Direct3D12 (NVIDIA RTX A500 Laptop GPU)) doesn't support base Zink requirements: feats.features.logicOp have_EXT_custom_border_color have_EXT_line_rasterization
glx: failed to create drisw screen
failed to load driver: zink
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Microsoft Corporation (0xffffffff)
    Device: D3D12 (Intel(R) Iris(R) Xe Graphics) (0xffffffff)
    Version: 24.0.3
    Accelerated: yes
    Video memory: 8146MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.1
    Max compat profile version: 4.1
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.0

< snip >

What happens without the LD_LIBRARY_PATH hack:

> glxinfo -B
name of display: :0
MESA: error: ZINK: failed to choose pdev
glx: failed to create drisw screen
failed to load driver: zink
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Mesa (0xffffffff)
    Device: llvmpipe (LLVM 17.0.6, 256 bits) (0xffffffff)
    Version: 24.0.3
    Accelerated: no
    Video memory: 7770MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 4.5
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2

< snip >

Where the ub folder has the following files:

> ls ub
libbsd.so.0
libedit.so.2
libmd.so.0
libstdc++.so.6
libtinfo.so.6
libz.so.1

WSL version

WSL version: 2.2.2.0
Kernel version: 5.15.150.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3296
SuperSandro2000 commented 5 months ago

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

Can you run the same commands with strace to see if it searches the right directories?

Pleune commented 5 months ago

Here is the output of MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia strace -o strace.log glxinfo -B, where llvmpipe is used instead of the GPU

strace.log

I can see many references to /run/opengl-driver/lib here, so it must be searching that path. Maybe the initial d3d12 library is being loaded, but then when the code inside that runs, dlopen fails?

SuperSandro2000 commented 5 months ago

It is not searching for d3d12 in the right location.

openat(AT_FDCWD, "/etc/drirc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/mitch/.drirc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/vcxkk3l25hlix62aivl2prfhpczrvymp-expat-2.6.2/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/zph9xw0drmq3rl2ik5slg0n2frw9lw5m-zlib-1.3.1/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/hqvkggqxqslgy2ahp1zswgh7im2kl5im-libxcb-1.16/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/qmy3imvwp6l129avxifnczw4p3fkdz0j-libX11-1.8.7/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/zmpw448943piz8vdav28j8dqfy54xwjc-libxshmfence-1.3.2/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ss6gh67xv5jw6jh0l7dwmyx9823wvb60-zstd-1.5.5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/blw10rx1cayp2n2pkmyihpipifzgj2xq-wayland-1.22.0/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/1birlqjpkhsadz9zxa1in8z1apz7l1x4-systemd-minimal-libs-255.4/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ixi9ca45786dq510fk6h7crmi9259qcd-libdrm-2.4.120/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/c2yb135iv4maadia5f760b3xhbh6jh61-gcc-13.2.0-lib/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/5xylmnrx48vqlcfv1daj63kql91gflx3-mesa-24.0.3-drivers/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/", 0x7ffdb4a032e0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/", 0x7ffdb4a032e0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
futex(0x7fe8aa4b4e1c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
openat(AT_FDCWD, "/nix/store/bhfs2c58cvgfmr5lh2j3p5032sy0bg11-mesa-24.0.3/share/drirc.d", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
getdents64(4, 0x186db00 /* 4 entries */, 32768) = 144
getdents64(4, 0x186db00 /* 0 entries */, 32768) = 0
close(4)                                = 0
openat(AT_FDCWD, "/nix/store/bhfs2c58cvgfmr5lh2j3p5032sy0bg11-mesa-24.0.3/share/drirc.d/00-mesa-defaults.conf", O_RDONLY) = 4
Pleune commented 5 months ago

hmm, libd3d12.so does exist in my nix store in from the wsl-lib and opengl-drivers packages:

> find /nix -name libd3d12.so
/nix/store/4mpn0189n0wrbi6xscricgsrdkx26bmx-wsl-lib/lib/libd3d12.so
/nix/store/gbfz16cazfd57f45654x916g06cwifxf-opengl-drivers/lib/libd3d12.so
/nix/store/1h8c6cx18mwn0i0kcz25mzbbf87mf8jp-opengl-drivers/lib/libd3d12.so
/nix/store/lk3pnpksmdw28m0nlj069vvg63l7ljjk-opengl-drivers/lib/libd3d12.so
/nix/store/0lxfb1j4yi3r0kaggq0hg0ad6q0i57ix-opengl-drivers/lib/libd3d12.so
/nix/store/57nb4gx6pmn46mvc8hlr26jmgm5j6i18-opengl-drivers/lib/libd3d12.so
/nix/store/6xq076h9n17fhb1w3f1pv1flspx1386m-opengl-drivers/lib/libd3d12.so
/nix/store/izbp5v8nvzs18gipcmd31iyywaqx10da-opengl-drivers/lib/libd3d12.so
/nix/store/v4wwzpgiazb98xjb0v4z0qpn4g2jwv84-opengl-drivers/lib/libd3d12.so
/nix/store/xzxjmcny8xdqv7b2sm0ws77zz65x9ylr-opengl-drivers/lib/libd3d12.so
/nix/store/10qydkiksz0zb9xn6fvlwr2s7swanzn1-opengl-drivers/lib/libd3d12.so
/nix/store/qq421zmhp9h5702mqlhirpxbf2jp5xb8-opengl-drivers/lib/libd3d12.so
/nix/store/hsd21kw4lpaxpivdyjrgz8c5vqm5iaq7-opengl-drivers/lib/libd3d12.so

I'm a good bit out of my depth here, but is there a way to add these packages as runtime dependencies for the mesa package?

Setting any of these directories as the LD_LIBRARY_PATH does allow at least the Nvidia d3d12 adapter to work

Pleune commented 5 months ago

Thank you, you have given my enough information to discover the hardware.opengl.setLdLibraryPath = true; option, which makes the Nvidia driver work. Perhaps it would be best to set this as default.

For MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel, I need to also get libedit and libtinfo into /run/opengl-drivers somehow. Perhaps this is a NixOS bug and not a NixOS on WSL bug.

Pleune commented 5 months ago

Alright, for a minimal configuration.nix only snippet, I have the following:

  hardware.opengl = {
    enable = true;
    driSupport = true;
    setLdLibraryPath = true;

    extraPackages = with pkgs; [
      mesa.drivers
      libvdpau-va-gl
      (libedit.overrideAttrs (attrs: {postInstall = (attrs.postInstall or "") + ''ln -s $out/lib/libedit.so $out/lib/libedit.so.2'';}))
    ];
  };

The libedit package only proides libedit.so.5 and libedit.so.0. d3d12 is looking for .2, so for the time being, this will work around that. It seems to pull in all of the other required library files as well.

sjhaleprogrammer commented 3 months ago 
Traceback (most recent call last):
  File "/home/samuel/git/stable-diffusion-webui/launch.py", line 48, in <module>
    main()
  File "/home/samuel/git/stable-diffusion-webui/launch.py", line 39, in main
    prepare_environment()
  File "/home/samuel/git/stable-diffusion-webui/modules/launch_utils.py", line 386, in prepare_environment
    raise RuntimeError(
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check
(venv)

how do i get my nvidia gpu to work

Parsifa1 commented 1 month ago

Thank you, you have given my enough information to discover the hardware.opengl.setLdLibraryPath = true; option, which makes the Nvidia driver work. Perhaps it would be best to set this as default.

For MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel, I need to also get libedit and libtinfo into /run/opengl-drivers somehow. Perhaps this is a NixOS bug and not a NixOS on WSL bug.

It seems that the option hardware.opengl.setLdLibraryPath = true; no longer exists

Pleune commented 1 month ago

@Parsifa1 Replace with 

environment.sessionVariables.LD_LIBRARY_PATH = [ "/run/opengl-driver/lib" ];

@SuperSandro2000, what do you think about adding this to the repo by default, particularly since this easier option is removed now? It is still needed on my system for my GPU to function, and I also don't think it would cause anyone else any issues. I would be happy to submit a PR with more documentation and explanation if you think that would be proper.

Perhaps we could even add options to enable the other paths I have had to added for Cuda and whatnot to work. Below is the full environment section I have been using now:

environment.sessionVariables = {
  CUDA_PATH = "${pkgs.cudatoolkit}";
  EXTRA_LDFLAGS = "-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib";
  EXTRA_CCFLAGS = "-I/usr/include";
  LD_LIBRARY_PATH = [
    "/usr/lib/wsl/lib"
    "${pkgs.linuxPackages.nvidia_x11}/lib"
    "${pkgs.ncurses5}/lib"
    "/run/opengl-driver/lib"
  ];
  MESA_D3D12_DEFAULT_ADAPTER_NAME = "Nvidia";
};
SuperSandro2000 commented 1 month ago

LD_LIBRARY_PATH is a big foot gun and can easily cause weird and hard to debug loading errors. I would love to really avoid it. If it is necessary we can add an option for it but I wouldn't turn it on by default.

The extra variables seem to be very user specific. I don't think they are a good fit.

polybluez commented 1 month ago

Maybe something like nixGL would be useful

Pleune commented 1 month ago

Maybe something like nixGL would be useful

I have actually tried this and it does not really change anything.

As to Sandro's comments, Everything other than "/run/opengl-driver/lib" is for the windows/wsl nvidia drivers to work, and to support cuda properly. It would be better to include the two pkgs additions in another way, like symlinking them into the same opengl lib folder so they don't end up effecting other parts of the system, but I was not able to get that to work.

polybluez commented 1 month ago

Maybe something like nixGL would be useful

I have actually tried this and it does not really change anything.

No, I mean having an utility like nixGL for running an app with WSL DirectX drivers or even adding support for it in nixGL

anticrisis commented 3 weeks ago

@SuperSandro2000 you said:

It is not searching for d3d12 in the right location.

I'm having the same ENOENT errors in strace when I launch emacs. I just installed NixOS on WSL in the past week or so, just trying to set everything up.

You also said:

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

but on my system, /usr/lib/wsl/lib is not symlinked. It contains all the cuda and d3d12 library files. And I apparently have no /run/opengl-driver/lib directory.

Pleune commented 3 weeks ago

@SuperSandro2000 you said:

It is not searching for d3d12 in the right location.

I'm having the same ENOENT errors in strace when I launch emacs. I just installed NixOS on WSL in the past week or so, just trying to set everything up.

You also said:

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

but on my system, /usr/lib/wsl/lib is not symlinked. It contains all the cuda and d3d12 library files. And I apparently have no /run/opengl-driver/lib directory.

You may have to reboot your wsl system to get the run folder back. For some reason mine is still unreliable, particularly after a laptop sleep.

But I have been looking more into exactly what files are needed to be linked into to keep WSLg happy, without causing any ld hack issues. To settle onto a "perfect solution" I'm considering trying changing the symlink from wsl/lib into a copy and elfpatch instead. I've had some success doing this manually now, but have not had the time to properly add it to the repo config and try it out.

anticrisis commented 3 weeks ago

You may have to reboot your wsl system to get the run folder back. For some reason mine is still unreliable, particularly after a laptop sleep.

wsl --shutdown wasn't doing it, but a reboot of the machine, did bring back the /run/opengl-driver/lib/ directory.

jacereda commented 3 weeks ago

Are you setting this?

https://github.com/nix-community/NixOS-WSL/blob/0bec2bfb8a2d4dd16e5b012982ca95e57d50e6a2/modules/wsl-distro.nix#L77

anticrisis commented 3 weeks ago

Are you setting this?

https://github.com/nix-community/NixOS-WSL/blob/0bec2bfb8a2d4dd16e5b012982ca95e57d50e6a2/modules/wsl-distro.nix#L77

No! Seems quite important. I only read the documentation, I should have read the code. I will give that a go.