cudaPackages: clean "cudatoolkit" up

[SomeoneSerge]

Suggested change

• Objective: ensure correctness (fix missing rpaths and broken wrappers); ensure that at least the runtime closures of downstream packages (like pytorchWithCuda) do not include gtk&c; ensure reasonably slim docker images
• [ ] Step 0: Rewrite cudatooklit to use autoPatchelf instead of the hard-coded $rpath. The autoPatchelf hook validates rpaths against DT_NEEDED, ensuring a higher level of correctness than we offer now
• [ ] Step 1: Move nsys-ui, cuda-gdb, or more (until no curses/gtk needed) out of cudatoolkit.out into separate outputs. The default output must not include the GUI or TUI apps
  • [ ] Step 2: Backport the change to nixos-21.11 s.t. last two releases can be used to build sensible docker images

• [ ] Step 3: Split dynamic and static outputs for redist cudaPackages.

  Rationale: if we were to transition e.g. pytorchWithCuda to redist packages now without fixing this first, its runtime closure would include static .a archives for cublas, et cetera. This is bogus behaviour.

• [ ] Step 3: Split the runfile-based cudatoolkit into multiple outputs analogous to redist cudaPackages. Add a way to populate cudaPackages with cudatoolkit contents, to be consumed following the same interface as redist packages. Add a utility function to merge selected outputs (cf. cudatoolkit_joined pattern in pytorch/tf expressions). Ensure that the utility function can be used both to merge either runfile-based cudatoolkit's outputs or the redist cudaPackages, using identical interface. This way expressions that have transitioned to redist packages can be rewired by users to build with cuda<11.4

Context

Nixpkgs has recently switched to the redist cuda packages (henceforth cudaPackages, as opposed to cudaPackages.cudatoolkit, or cudatoolkit for short), where every individual piece of cuda comes in a separate derivation. The old run-file based cudatoolkit expression, which packs development libraries and GUI apps into the same output, is being phased out. However, transition of downstream packages is slow and we still heavily rely on that old expression. This has far extending ramifications:

• Runtime closures of packages like pytorchWithCuda needlessly include ALSA, Xorg, fontconfig, &c libraries. Importantly, dockerTools.buildLayeredImage packing pytorchWithCuda necessarily includes these libraries too:
  • Cf. runtime and build closures
  • Cf. contents of a docker image shipping pytorchWithCuda
• Heavy packages like pytorchWithCuda and tensorflowWithCuda needlessly rebuild something like fontconfig updates

The cudatoolkit expression at some point was split into cudatoolkit.out and cudatoolkit.lib exactly for these reasons. However, the existing .lib output only includes cudart which is insufficient to build nearly anything (usually packages additionally ask at least for cublas, cufft, curand). Downstream nixpkgs expressions consume cudatoolkit.out instead, consistently.

Besides, the current expression is already buggy:

• Some binaries have missing rpaths
• Some wrappers are broken (e.g. /nix/store/s1dr6vd3f2wvvxf2zynszga69znpqnyh-cudatoolkit-11.6.1/bin/.nvvp-wrapped: line 3: /nix/store/s1dr6vd3f2wvvxf2zynszga69znpqnyh-cudatoolkit-11.6.1/bin/../libnvvp/nvvp: No such file or directory)

In this issue I suggest we focus on the very first item, i.e. the docker images and runtime closures (opposed to frequent rebuilds). Simply splitting nsys and debuggers into their own outputs should amend the issue.

I suggest the change be backported into the previous release

• because nsys and cuda-gdb are leaf programs used interactively, i.e. their few users can easily adjust;
• I consider the current situation (gtk&cups finding their way into the docker image) to be bogus, affecting the previous release too;
• people (e.g. I) may be building images with the previous release nixpkgs too, to access older versions of downstream packages.

Unaddressed issues, alternatives

With these changes applied, the cudatoolkit.out.outPath computed from the inputs would still change with every fontconfig update, although the contents shouldn't: we're still going to rebuild. Content-addressable mode could amend this?

As an alternative I've tried mass-switching packages to cudatoolkit.lib, but that wasn't meaningful without copying more things into $lib first

CC @NixOS/cuda-maintainers @FRidh

[FRidh]

What is the reason not more packages can be converted to the redist packages? Are parts from cudatoolkit missing? Splitting/removing static outputs is trivial.

[SomeoneSerge]

What is the reason not more packages can be converted to the redist packages? Are parts from cudatoolkit missing?

We'll keep transitioning packages to the redist cuda. This is a slower process, but we can approach the desired goal (for the scope of this issue: "slim docker images") more gradually, by first pruning the old expression a little. That should be pretty easy to merge. That we can also backport, probably unlike the redist packages.

Splitting static is trivial, but establishing a common interface for runfile and redist is not. That is still desirable so that users may switch to older (<11.4) cuda when needed, until these cuda versions have been fully deprecated by the upstream.

Obviously, there's a hierarchy of priorities and just splitting the outputs can and should come before we've agreed on a common interface

[samuela]

I don't use docker personally, but I think the proposed changes make sense. We should always be aiming to keep things as slim as possible IMHO.