[Bug] linuxPackages_cachyos-lto.nvidia_x11 fails to build (fix included)

[silvanshade]

What happens?

The linuxPackages_cachyos-lto.nvidia_x11 fails to build due to permission errors related to ThinLTO.

One would encounter this with the following configuration:

{
  boot.kernelPackages = pkgs.linuxPackages_cachyos-lto;
  hardware.nvidia.package = config.boot.kernelPackages.nvidia_x11;
}

What is expected to happen?

Ideally linuxPackages_cachyos-lto.nvidia_x11 should build without errors.

If possible, please attach logs

No response

More information

It's possible to fix the ThinLTO permission errors by using the --thinlto-cache-dir=<dir> option and specifying a directory with appropriate permissions.

For example, you can create a suitable directory with the following:

sudo mkdir -p -m0770 /var/cache/clang-thinlto
sudo chown root:nixbld /var/cache/clang-thinlto

Next you need to add the path to the build sandbox paths and override the makeFlags of the kernel:

{
  nix.settings.extra-sandbox-paths = [
    "/var/cache/clang-thinlto"
  ];
  nixpkgs.overlays = [
    (final: prev: {
      linuxPackages_cachyos-lto-with-fixed-nvidia = final.linuxPackages_cachyos-lto.override (linuxPackagesArgs: {
        linuxPackagesFor = kernel: linuxPackagesArgs.linuxPackagesFor (
          kernel.overrideAttrs (kernelAttrs: {
            makeFlags = kernelAttrs.makeFlags ++ [
              "KBUILD_LDFLAGS+=--thinlto-cache-dir=/var/cache/clang-thinlto"
            ];
          })
        );
      });
    })
  ];
  boot.kernelPackages = pkgs.linuxPackages_cachyos-lto-with-fixed-nvidia;
  hardware.nvidia.package = config.boot.kernelPackages.nvidia_x11;
}

[silvanshade]

Maybe it would be reasonable to add special handling for the nvidia driver in the LTO case to automatically set this ThinLTO cache dir in order to avoid build failures?

I don't know if this could work with the sandbox modification approach (can you change sandbox paths per-derivation?) but it should be enough to create a temporary directory (with a name chosen at build-time for the kernel derivation).

It wouldn't take advantage of any caching that way but that's not really important. The main thing is that using this setting prevents the nvidia build script from trying to write to the existing kernel modules path which fails due to permission errors.

[PedroHLC]

For starters, add it to this list: https://github.com/chaotic-cx/nyx/blob/8cb2e4eb3422d099ce80c95accfbba917c45622d/pkgs/linux-cachyos/default.nix#L42

This triggers the fixes from: https://github.com/chaotic-cx/nyx/blob/8cb2e4eb3422d099ce80c95accfbba917c45622d/pkgs/linux-cachyos/lib/llvm-module-overlay.nix#L8

Edit: Also, I've seen users in the past just replacing it with ordinary linuxPackagesFor_cachyos.nvidia-x11 and it worked (you have to use .extend in the Packages attrset, no need for an overlay, you can do it directly in the kernel option)

[silvanshade]

For starters, add it to this list:

https://github.com/chaotic-cx/nyx/blob/8cb2e4eb3422d099ce80c95accfbba917c45622d/pkgs/linux-cachyos/default.nix#L42

This triggers the fixes from:

https://github.com/chaotic-cx/nyx/blob/8cb2e4eb3422d099ce80c95accfbba917c45622d/pkgs/linux-cachyos/lib/llvm-module-overlay.nix#L8

Oh interesting, I'll try that. Thanks.

Edit: Also, I've seen users in the past just replacing it with ordinary linuxPackagesFor_cachyos.nvidia-x11 and it worked (you have to use .extend in the Packages attrset, no need for an overlay, you can do it directly in the kernel option)

Yeah this does work and I was using it for awhile. Unfortunately in my case this results in two builds of the kernel since I use the znver4 package from the AMD64 microarchitectures package set (and I sometimes add additional KCFLAGS). So a solution that didn't require that would be nice.

[diniamo]

This happens for xpadneo as well:

hardware.xpadneo.enable = true;

And the fix doesn't seem to work for me.

[PedroHLC]

I can try to fix these two during the 6.9.0 bump tomorrow.

[diniamo]

@PedroHLC any update on this?