search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
16.51k stars 12.99k forks source link

Sunshine nvenc error `Cannot load libcuda.so.1`, defaulting to software encoding #272221

Open PierreBeucher opened 7 months ago

PierreBeucher commented 7 months ago

Describe the bug

Upgrading from NixOS 23.05 => 23.11 caused Sunshine upgrade 0.19.x => 0.21.0 , but nvenc and vaapi encoding now fails with Cannot load libcuda.so.1 and similar errors, eventually falling back to software encoding. These errors didn't show up before.

It may be something similar to https://github.com/NixOS/nixpkgs/issues/271333 (solved by https://github.com/NixOS/nixpkgs/pull/271352) which I also hit and solved locally by using 

    sunshineOverride = pkgs.sunshine.overrideAttrs (prev: {
        runtimeDependencies = prev.runtimeDependencies ++ [ pkgs.libglvnd ];
    });

But the problem persist. Maybe there's another runtimeDependencies requiring libcuda.so.1 somehow?

For context:

Here are full log outputs:

Started Sunshine server.
[origin_web_ui_allowed] -- [lan]
[2023:12:03:20:04:39]: Info: Sunshine version: 0.21.0
[2023:12:03:20:04:39]: Error: Environment variable WAYLAND_DISPLAY has not been defined
[2023:12:03:20:04:39]: Info: Detecting monitors
[2023:12:03:20:04:39]: Info: Detected monitor 0: DVI-D-0, connected: true
[2023:12:03:20:04:39]: Info: Detected monitor 1: DVI-D-1, connected: false
[2023:12:03:20:04:39]: Info: Detected monitor 2: DVI-D-2, connected: false
[2023:12:03:20:04:39]: Info: Detected monitor 3: DVI-D-3, connected: false
[2023:12:03:20:04:39]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2023:12:03:20:04:39]: Info: Trying encoder [nvenc]
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x33f7240] Cannot load libcuda.so.1
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x33f7240] Could not dynamically load CUDA
[2023:12:03:20:04:39]: Error: Failed to create a CUDA device: Operation not permitted
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: System tray created
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x3403580] Cannot load libcuda.so.1
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x3403580] Could not dynamically load CUDA
[2023:12:03:20:04:39]: Error: Failed to create a CUDA device: Operation not permitted
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x340d040] Cannot load libcuda.so.1
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x340d040] Could not dynamically load CUDA
[2023:12:03:20:04:39]: Error: Failed to create a CUDA device: Operation not permitted
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x334ea40] Cannot load libcuda.so.1
[2023:12:03:20:04:39]: Error: [AVHWDeviceContext @ 0x334ea40] Could not dynamically load CUDA
[2023:12:03:20:04:39]: Error: Failed to create a CUDA device: Operation not permitted
[2023:12:03:20:04:39]: Info: Encoder [nvenc] failed
[2023:12:03:20:04:39]: Info: Trying encoder [vaapi]
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
libva info: VA-API version 1.20.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[2023:12:03:20:04:39]: Error: [h264_vaapi @ 0x334e080] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2023:12:03:20:04:39]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
libva info: VA-API version 1.20.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[2023:12:03:20:04:39]: Error: [h264_vaapi @ 0x334e080] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2023:12:03:20:04:39]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
libva info: VA-API version 1.20.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[2023:12:03:20:04:39]: Error: [h264_vaapi @ 0x334e080] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2023:12:03:20:04:39]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
libva info: VA-API version 1.20.0
libva info: Trying to open /run/opengl-driver/lib/dri/nvidia_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[2023:12:03:20:04:39]: Error: [h264_vaapi @ 0x334e080] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2023:12:03:20:04:39]: Error: Could not open codec [h264_vaapi]: Function not implemented
[2023:12:03:20:04:39]: Info: Encoder [vaapi] failed
[2023:12:03:20:04:39]: Info: Trying encoder [software]
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: Screencasting with X11
[2023:12:03:20:04:39]: Info: SDR color coding [Rec. 601]
[2023:12:03:20:04:39]: Info: Color depth: 8-bit
[2023:12:03:20:04:39]: Info: Color range: [JPEG]
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] profile High, level 4.2, 4:2:0, 8-bit
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] frame I:1     Avg QP: 6.10  size:  1182
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] mb I  I16..4: 100.0%  0.0%  0.0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] 8x8 transform intra:0.0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] coded y,uvDC,uvAC intra: 0.0% 0.0% 0.0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] i16 v,h,dc,p: 99%  0%  1%  0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 50%  0% 50%  0%  0%  0%  0%  0%  0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] i8c dc,h,v,p: 100%  0%  0%  0%
[2023:12:03:20:04:39]: Info: [libx264 @ 0x334e080] kb/s:567.36
[2023:12:03:20:04:39]: Info:
[2023:12:03:20:04:39]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2023:12:03:20:04:39]: Info:
[2023:12:03:20:04:39]: Info: Found H.264 encoder: libx264 [software]
[2023:12:03:20:04:39]: Info: Adding avahi service Sunshine
[2023:12:03:20:04:39]: Error: Failed to add _nvstream._tcp service: Not permitted
[2023:12:03:20:04:39]: Info: Configuration UI available at [https://localhost:47990]

I suppose I'm missing some OS packages, but I don't know which ones as NVidia drivers and CUDA are installed and were working fine before.

Expected behavior

Expect nvenv encoding to work without libcuda.so.1 lodading errors

Notify maintainers

@devusb you seem quite knowledgeable on the subject, maybe you can help? Thanks

Add a :+1: reaction to issues you find important.

PierreBeucher commented 7 months ago

I found an option cudaSupport in sunshine/default.nix tried to enable cuda, tried to enable it but it failed:

nixpkgs.config.cudaSupport = true;

Log outputs:

# ...
cmake flags: -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/share/doc/Sunshine -DCMAKE_INSTALL_INFODIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/share/info -DCMAKE_INSTALL_MANDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/share/man -DCMAKE_INSTALL_OLDINCLUDEDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/include -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/include -DCMAKE_INSTALL_SBINDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/sbin -DCMAKE_INSTALL_BINDIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_OSX_SYSROOT= -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/v3p0rlln4cv1hkp506jsn3f1jg3p52n7-binutils-wrapper-2.40/bin/strip -DCMAKE_RANLIB=/nix/store/v3p0rlln4cv1hkp506jsn3f1jg3p52n7-binutils-wrapper-2.40/bin/ranlib -DCMAKE_AR=/nix/store/v3p0rlln4cv1hkp506jsn3f1jg3p52n7-binutils-wrapper-2.40/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=/nix/store/cn1qhyk74nbz21s00rzf6awnyifqnqid-sunshine-0.21.0 -Wno-dev -DCUDA_TOOLKIT_ROOT_DIR=/nix/store/jcmaarfa2mk9qpjipihmk0h4g6my6wnx-cudatoolkit-11.8.0 -DCUDA_HOST_COMPILER=/nix/store/b07vs605q0sn31jnhr1a06mrcmxppl4f-gcc-wrapper-11.4.0/bin/c++ -DCMAKE_CUDA_HOST_COMPILER=/nix/store/b07vs605q0sn31jnhr1a06mrcmxppl4f-gcc-wrapper-11.4.0/bin/c++ -DCUDAToolkit_INCLUDE_DIR=/nix/store/jcmaarfa2mk9qpjipihmk0h4g6my6wnx-cudatoolkit-11.8.0/include -DCUDAToolkit_ROOT=/nix/store/jcmaarfa2mk9qpjipihmk0h4g6my6wnx-cudatoolkit-11.8.0;/nix/store/i0xr8ycrbj6azsvyb6v4qj9fml5yzhq8-cudatoolkit-11.8.0-lib 
-- The C compiler identification is GNU 12.3.0
-- The CXX compiler identification is GNU 12.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /nix/store/90h6k8ylkgn81k10190v5c9ldyjpzgl9-gcc-wrapper-12.3.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /nix/store/90h6k8ylkgn81k10190v5c9ldyjpzgl9-gcc-wrapper-12.3.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find Git (missing: GIT_EXECUTABLE) 
CMake Warning at cmake/prep/build_version.cmake:56 (MESSAGE):
  : Git not found, cannot find git version
Call Stack (most recent call first):
  CMakeLists.txt:26 (include)

-- Looking for fcntl
-- Looking for fcntl - found
-- Looking for ioctl
-- Looking for ioctl - found
-- Looking for poll
-- Looking for poll - found
-- Looking for getaddrinfo
-- Looking for getaddrinfo - found
-- Looking for getnameinfo
-- Looking for getnameinfo - found
-- Looking for gethostbyname_r
-- Looking for gethostbyname_r - found
-- Looking for gethostbyaddr_r
-- Looking for gethostbyaddr_r - found
-- Looking for inet_pton
-- Looking for inet_pton - found
-- Looking for inet_ntop
-- Looking for inet_ntop - found
-- Performing Test HAS_MSGHDR_FLAGS
-- Performing Test HAS_MSGHDR_FLAGS - Success
-- Check size of socklen_t
-- Check size of socklen_t - done
-- Check size of QOS_FLOWID
-- Check size of QOS_FLOWID - failed
-- Check size of PQOS_FLOWID
-- Check size of PQOS_FLOWID - failed
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found Boost: /nix/store/x1pywcr1cql9r271sl4m6h882hv2mczc-boost-1.81.0-dev/lib/cmake/Boost-1.81.0/BoostConfig.cmake (found suitable version "1.81.0", minimum required is "1.53.0") found components: system thread 
-- Found OpenSSL: /nix/store/k3d7ny5h682kixy91iw7sm653kxyqr8d-openssl-3.0.12/lib/libcrypto.so (found version "3.0.12")  
-- Found PkgConfig: /nix/store/ljrkf0kwgckcy02kdlz8sjdnxb2pwjb4-pkg-config-wrapper-0.29.2/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'libcurl'
--   Found libcurl, version 8.4.0
-- Found Boost: /nix/store/x1pywcr1cql9r271sl4m6h882hv2mczc-boost-1.81.0-dev/lib/cmake/Boost-1.81.0/BoostConfig.cmake (found version "1.81.0") found components: locale log filesystem program_options 
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - /nix/store/jcmaarfa2mk9qpjipihmk0h4g6my6wnx-cudatoolkit-11.8.0/bin/nvcc
-- Looking for a CUDA host compiler - /nix/store/b07vs605q0sn31jnhr1a06mrcmxppl4f-gcc-wrapper-11.4.0/bin/c++
-- The CUDA compiler identification is NVIDIA 11.8.89
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /nix/store/jcmaarfa2mk9qpjipihmk0h4g6my6wnx-cudatoolkit-11.8.0/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- CUDA Compiler Version: 11.8.89
-- CUDA Architectures: 35;50;52;60;61;62;70;75;80;86;90
-- Checking for module 'libdrm'
--   Found libdrm, version 2.4.117
-- Found LIBDRM: /nix/store/bvg3cgclpcz6qa8p4l3pj8paqwikiksq-libdrm-2.4.117/lib/libdrm.so  
-- Checking for module 'libcap'
--   Found libcap, version 2.69
-- Found LIBCAP: /nix/store/8bzgq1kprz2cxar7kvhscjgqnjhbd0i8-libcap-2.69-lib/lib/libcap.so  
-- Found Wayland: /nix/store/y3i4dfxcs1blghaj4bvdxng6la44l8rz-wayland-1.22.0/lib/libwayland-client.so;/nix/store/y3i4dfxcs1blghaj4bvdxng6la44l8rz-wayland-1.22.0/lib/libwayland-server.so;/nix/store/y3i4dfxcs1blghaj4bvdxng6la44l8rz-wayland-1.22.0/lib/libwayland-egl.so;/nix/store/y3i4dfxcs1blghaj4bvdxng6la44l8rz-wayland-1.22.0/lib/libwayland-cursor.so   
wayland-scanner private-code /build/source/third-party/wayland-protocols/unstable/xdg-output/xdg-output-unstable-v1.xml /build/source/build/generated-src/xdg-output-unstable-v1.c
wayland-scanner client-header /build/source/third-party/wayland-protocols/unstable/xdg-output/xdg-output-unstable-v1.xml /build/source/build/generated-src/xdg-output-unstable-v1.h
wayland-scanner private-code /build/source/third-party/wlr-protocols/unstable/wlr-export-dmabuf-unstable-v1.xml /build/source/build/generated-src/wlr-export-dmabuf-unstable-v1.c
wayland-scanner client-header /build/source/third-party/wlr-protocols/unstable/wlr-export-dmabuf-unstable-v1.xml /build/source/build/generated-src/wlr-export-dmabuf-unstable-v1.h
-- Found X11: /nix/store/nzw7krhdm0ijbfmskvqrzw9nabqygv18-xorgproto-2023.2/include   
-- Looking for XOpenDisplay in /nix/store/b7pb0mmcwjp973nj8gcaa2fnmqxckc8q-libX11-1.8.7/lib/libX11.so;/nix/store/qyyw5j9xn5mnjlv5f469h12clp4wdj7b-libXext-1.3.5/lib/libXext.so
-- Looking for XOpenDisplay in /nix/store/b7pb0mmcwjp973nj8gcaa2fnmqxckc8q-libX11-1.8.7/lib/libX11.so;/nix/store/qyyw5j9xn5mnjlv5f469h12clp4wdj7b-libXext-1.3.5/lib/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Checking for module 'appindicator3-0.1'
--   Found appindicator3-0.1, version 12.10.0
-- Checking for module 'libnotify'
--   Found libnotify, version 0.8.3
-- Configuring done (15.1s)
-- Generating done (0.0s)
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_EXPORT_NO_PACKAGE_REGISTRY
    CMAKE_POLICY_DEFAULT_CMP0025
    CUDAToolkit_INCLUDE_DIR
    CUDAToolkit_ROOT
    CUDA_HOST_COMPILER
    CUDA_TOOLKIT_ROOT_DIR

-- Build files have been written to: /build/source/build
cmake: enabled parallel building
cmake: enabled parallel installing
building
build flags: -j4 SHELL=/nix/store/q1c2flcykgr4wwg5a6h450hxbk4ch589-bash-5.2-p15/bin/bash
[  6%] Building C object third-party/miniupnp/miniupnpc/CMakeFiles/libminiupnpc-static.dir/src/miniupnpc.c.o
# [...]
[100%] Linking CXX executable sunshine
/nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: /nix/store/2gjifw9iylbkiv56jyvfi2ybyc0ivibi-icu4c-73.2/lib/libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/sunshine.dir/build.make:772: sunshine-0.21.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:137: CMakeFiles/sunshine.dir/all] Error 2
make: *** [Makefile:156: all] Error 2
error: builder for '/nix/store/3n6hbprp7rs4zx0yp4i3mb4zr0qahqcp-sunshine-0.21.0.drv' failed with exit code 2
PierreBeucher commented 7 months ago

Found a workaround by adding a package exposing libcuda.so.1 to runtimeDependencies, though I'm not sure it's the proper way: 

sunshineOverride = pkgs.sunshine.overrideAttrs (prev: {
        runtimeDependencies = prev.runtimeDependencies ++ [ 
            pkgs.linuxKernel.packages.linux_zen.nvidia_x11
        ];
    });
devusb commented 7 months ago

I don't have a (reliable) NVIDIA box anymore, but I think https://github.com/NixOS/nixpkgs/pull/235655 should resolve building with config.cudaSupport, it'll just take a rebase.

I would think that autoAddOpenGLRunpathHook could expose libcuda.so.1 if it were added with config.cudaSupport -- maybe give that a shot and see if it helps?

SomeoneSerge commented 7 months ago

Hi! The libcuda.so refers to nvidia's "userspace driver" library, which does not work unless it matches exactly the version of the kernel module loaded at boot time. This is why it comes from the nvidia_x11 package. We never link nvidia_x11 directly because that's useless on any machine that doesn't run the exact same kernel. This is more or less just a choice made by nvidia

This is why we link libcuda.so impurely using the addDriverRunpath hook (addOpenGLRunpath is the legacy name). What this does is it adds a record in the binaries' headers for the dynamic linker to look the dependencies up in one additional "impure" location (currently it's /run/opengl-driver/lib) where NixOS deploys its userspace drivers at runtime. On FHS distributions (e.g. Ubuntu) one would have to just use an extra tool to help the dynamic linker locate the host system's driver (e.g. https://github.com/numtide/nix-gl-host).

The config.cudaSupport option refers to Nixpkgs instead and it does not handle the driver. What it does, roughly, is it tells Nixpkgs to enable building the CUDA kernels even when they're optional. E.g. (import nixpkgs {}).python3Packages.torch can't use CUDA even if you have the driver, because you need to build the kernels and link some of the cudatoolkit's libraries. For that you use (import nixpkgs { config.cudaSupport = true; }).python3Packages.torch.

PierreBeucher commented 6 months ago

Thanks for your replies ! I'll give it a try as soon as I have a few moments.

SomeoneSerge commented 6 months ago

Hey, now that I read this again, this looks like it might be a libstdc++ compatibility issue:

[100%] Linking CXX executable sunshine
/nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: /nix/store/2gjifw9iylbkiv56jyvfi2ybyc0ivibi-icu4c-73.2/lib/libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'

The current workaround is: https://github.com/NixOS/nixpkgs/blob/e3f251c662bc525e4bae3edfa3fc67e52d690d4f/pkgs/development/libraries/opencv/4.x.nix#L90-L93

devusb commented 6 months ago

https://github.com/NixOS/nixpkgs/pull/235655/files this MR adds the workaround -- maybe worth a shot to see if it resolves this issue.

SomeoneSerge commented 6 months ago

@devusb fwiw,

❯ nix build -I nixpkgs=flake:github:NixOS/nixpkgs/master -f '<nixpkgs>' --arg config '{ cudaSupport = true; cudaCapabilities = [ "8.6" ]; cudaEnableForwardCompat = false; allowUnfree = true; }' -L sunshine --print-out-paths 
/nix/store/fwqg9qb6cn5sczacc0gy9scnyi4bh8fh-sunshine-0.21.0

works fine, so maybe that was just a nixos-unstable issue at the time

witchof0x20 commented 5 months ago

Just chiming in to say I'm having this issue as well (23.11 branch). Overriding cudasupport/stdenv does not fix it for me.

SomeoneSerge commented 5 months ago

@witchof0x20 is it the libstdc++ or the libcuda error that you're getting? In case of the former, the comment regarding stdenv still applies: https://github.com/NixOS/nixpkgs/blob/f4a8d6d5324c327dcc2d863eb7f3cc06ad630df4/pkgs/servers/sunshine/default.nix#L46

peperunas commented 3 months ago

Should have been fixed by #293938

PierreBeucher commented 3 months ago

Awesome thanks, will try again ASAP

therealcoochieman commented 3 months ago

Hello, I'm just letting you know that the Stdenv fix still results in libcuda not being loaded. I am overriding sunshine as to enable cudaSupport and cuda backendStdenv... Is there something I am missing?

SomeoneSerge commented 2 months ago

Hello, I'm just letting you know that the Stdenv fix still results in libcuda not being loaded.

Please elaborate: describe how to reproduce your configuration, attach the logs with LD_DEBUG=libs