search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
16.46k stars 12.95k forks source link

python3Packages.torchWithCuda: lazy loading nvrtc #296179

Open SomeoneSerge opened 3 months ago

SomeoneSerge commented 3 months ago

Describe the bug

Pytorch uses dlopen to load libnvrtc.so without publicly declaring a dependency on it: https://github.com/pytorch/pytorch/blob/014f91a9d9f94ac9a7f0711600240d7cd7f69844/aten/src/ATen/cuda/detail/LazyNVRTC.cpp#L78-L83. No declaration -> no runpaths preserved after installation -> dlopen fails.

Steps To Reproduce

Steps to reproduce the behavior:

  1. nix run github:SomeoneSerge/dust3r.nix/cf159ccbab05b6b023b8f5050572adfa21b30a34#pretrained;
  2. open the web page;
  3. feed 2+ images;
  4. Observe a runtime error
  File "/nix/store/c64592rcivfc0w373nfh41j9lshb0w3s-python3.11-roma-unstable-2024-02-26/lib/python3.11/site-packages/roma/mappings.py", line 77, in procrustes
    R, DS = _ProcrustesManualDerivatives.apply(M, force_rotation, regularization, gradient_eps)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/n33zdaqxa8j711plws8xyjpzdnw0idmr-python3.11-torch-2.2.1/lib/python3.11/site-packages/torch/autograd/function.py", line 553, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/c64592rcivfc0w373nfh41j9lshb0w3s-python3.11-roma-unstable-2024-02-26/lib/python3.11/site-packages/roma/mappings.py", line 22, in forward
    flip = (torch.det(U) * torch.det(V) < 0)
    ^^^^^^^^^^^^
RuntimeError: Error in dlopen for library libnvrtc.so.12and libnvrtc-26bfb51d.so.12

Additional context

Add any other context about the problem here.

Notify maintainers

@NixOS/cuda-maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

Add a :+1: reaction to issues you find important.

hoh commented 3 months ago

I am facing the same issue when using transformers, thanks for looking into this !

SomeoneSerge commented 3 months ago

I am facing the same issue when using transformers, thanks for looking into this !

You can already use the version in the PR: I just need to rewrite because I broke some jetson-related stuff. Alternatively, the overlay in https://github.com/SomeoneSerge/dust3r.nix/commit/8715f78a6f875fa5a1795da8f56e05fe2b2811d6 should work too