Hashcat with CUDA is broken

[Janrupf]

Describe the bug

Attempting to run hashcat with CUDA results in the CUDA stub driver being loaded because /run/opengl-driver/lib is not on LD_LIBRARY_PATH.

Steps To Reproduce

Steps to reproduce the behavior:

1. enable nixpkgs.config.cudaSupport = true
2. use the hashcat package
3. run hashcat

Expected behavior

/run/opengl-driver/lib should be added to the LD_LIBRARY_PATH automatically so that hashcat runs.

Screenshots

> hashcat hash.txt
hashcat (6.2.6) starting in autodetect mode

cuDriverGetVersion(): 34

Started: Thu May  9 01:59:22 2024
Stopped: Thu May  9 01:59:22 2024

Additional context

The package seems to try to add the OpenGL run path, but it doesn't work (for some reason?).

See here: https://github.com/NixOS/nixpkgs/blob/9a35766bd30c07f39c881fc439db65ee1fff65b2/pkgs/tools/security/hashcat/default.nix#L71-L76

Notify maintainers

@felixalbrigtsen @kierdavis @zimbatm

Metadata

> nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.29, NixOS, 24.05 (Uakari), 24.05.20240502.63c3a29`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - nixpkgs: `/nix/store/p69bcs7ma6ijj8v9xsrg3nq3nn8ryn95-source`

---

Add a :+1: reaction to issues you find important.

[felixalbrigtsen]

I believe it is the same problem that is seen and explained in https://github.com/NixOS/nixpkgs/issues/272221#issuecomment-1848816792, and agree that only the stub driver is loaded. My computer with proper Nvidia drivers is unavailable at the moment, so I can't test it out right now, but I have some ideas:

Have you enabled the graphics drivers like described at https://nixos.wiki/wiki/Nvidia (or similar documentation for your type of graphics card), and does it seem to work in other applications that use the GPU?

Does it work if you hard code that library path, something like the diff below?

diff --git a/pkgs/tools/security/hashcat/default.nix b/pkgs/tools/security/hashcat/default.nix
index 56399e61639e..9a239c0b7e3a 100644
--- a/pkgs/tools/security/hashcat/default.nix
+++ b/pkgs/tools/security/hashcat/default.nix
@@ -64,6 +64,7 @@ stdenv.mkDerivation rec {
       "${ocl-icd}/lib"
     ] ++ lib.optionals cudaSupport [
       "${cudaPackages.cudatoolkit}/lib"
+      "/run/opengl-driver/lib"
     ]);
   in ''
     wrapProgram $out/bin/hashcat \

[Janrupf]

Nope, that doesn't fix the issue - but almost. Changing the order in which the paths are added does actually fix the issue, so this works:

ib.optionals cudaSupport [
  "/run/opengl-driver/lib" # Note that this needs to be first
  "${cudaPackages.cudatoolkit}/lib"
]);

(also yes, GPU drivers are set up correctly and working as intended, including CUDA)

When looking at the duplicated files in both directories, the issue seems to stem from the fact that the way the library path is set up.

$ for f in $(ls "$(NIXPKGS_ALLOW_UNFREE=1 nix eval --impure --raw nixpkgs#cudatoolkit)/lib"); do if [[ -f "/run/opengl-driver/lib/$f" ]]; then echo "Duplicated: $f"; fi; done
Duplicated: libcuda.so
Duplicated: libcuda.so.1

I guess if it loads libcuda.so from the nixpkgs#cudatoolkit path then it finds the stub driver, but if it loads from /run/opengl-driver/lib it obviously finds and loads the currently running driver.

[felixalbrigtsen]

Thank you for a good report!

From the definition of addOpenGLRunpath, I think the current hashcat package should have worked as is, but I am not really sure if it's a bug or if I got something wrong. It looks like at least some other applications runaddOpenGLRunpath on the shared libraries: https://github.com/NixOS/nixpkgs/blob/47b604b07d1e8146d5398b42d3306fdebd343986/pkgs/development/python-modules/tensorflow/default.nix#L565-L569 We only run it on the one executable, but our hashcat libraries in $out/share don't seem to reference libcuda directly(?).

Alternatively, some programs, like https://github.com/NixOS/nixpkgs/blob/e5a9c059efe2fae982ffdd555d9b7ae677fde76f/nixos/modules/services/misc/plex.nix#L179 just put opengl-drivers in the LD_LIBRARY_PATH like we did in the comments above. Maybe it would be okay to simply make change you tested permanent? I don't think I see a problem with "hard coding" /run/opengl-driver/lib like this.

[eclairevoyant]

fyi addDriverRunpath is the current preference over addOpenGLRunpath

https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/cuda.section.md may be helpful as well

[SomeoneSerge]

I think this is a regression from https://github.com/NixOS/nixpkgs/pull/301188 and https://github.com/NixOS/nixpkgs/pull/302845. Hashcat puts ${cudaPackages.cudatoolkit}/lib into LD_LIBRARY_PATH (this is bad btw, because LD_LIBRARY_PATH has higher priority than DT_RUNPATH which is correctly prefixed with ${addDriverRunpath.driverLink}/lib). With the first PR, cudatoolkit becomes a symlinkJoin of a bunch of semi-automatically selected paths. With the second, a symlink to the stub slipped in at ${cudatoolkit}/lib/libcuda.so because the get{Dev,Lib,Static} cuda_cudart now all resolve into the cuda_cudart.out, which we absolutely must delete contains symlinks to all outputs, and that includes cuda_cudart.stubs. CC @NixOS/cuda-maintainers

[SomeoneSerge]

As a reminder, the fix is to add ${getLib cuda_cudart}/lib to DT_RUNPATH.

RE: "revert getOutput" The change was reintroduced in https://github.com/NixOS/nixpkgs/pull/323056