Large scale dependency exclusion and more robust custom indices behavior

[Immortalin]

When installing encodec, I wanted to install it in CPU-only mode due to lack of an nvidia processor on my local machine. It required going through the generated lock file and manually excluding around 10 nvidia dependencies, even when custom indices have been added. The indice resolution should happen before the dependency locking step.

dependencies = ["encodec~=0.1.1", "torch~=2.0.1", "torchvision~=0.15.2", "torchaudio~=2.0.2"]
readme = "README.md"
requires-python = ">= 3.8"

[[tool.rye.sources]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
type = "find-links"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.rye]
managed = true
dev-dependencies = ["black~=23.3.0", "ruff~=0.0.270"]
excluded-dependencies = ["nvidia-cublas-cu11", "nvidia-cuda-cupti-cu11", "nvidia-cuda-nvrtc-cu11", "nvidia-cuda-runtime-cu11", "nvidia-cudnn-cu11", "nvidia-cufft-cu11", "nvidia-curand-cu11", "nvidia-cusolver-cu11", "nvidia-cusparse-cu11", "nvidia-nccl-cu11", "nvidia-nvtx-cu11"]
[tool.hatch.metadata]
allow-direct-references = true

It was not sufficient to blacklist nvidia dependencies. When the import torch command was given to the python interpreter, the following error occured:

  File "/home/user/.rye/py/cpython@3.11.3/install/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libcurand.so.10: cannot open shared object file: No such file or directory

[ischaojie]

Not sure about your environment, when I use the same configuration as above no error occurs: rye run python

>>> import torch 
>>> 

(Update):

rye 0.6.0
commit: unknown (1731a5b37 2023-06-01)
platform: macos (aarch64)
self-python: cpython@3.10
symlink support: true

[CharlesChen0823]

i repro in my machine(20.04.1-Ubuntu), current not find the solution.

[mitsuhiko]

@CharlesChen0823 can you paste the output of rye --version?

[CharlesChen0823]

i repro with latest version

rye 0.6.0 commit: 0.5.0+93 (53cf0df44 2023-06-02) platform: linux (x86_64) self-python: cpython@3.10 symlink support: true

with the error exception

charles@charles:~/program/python/test_nvidia$ rye run python
Python 3.8.16 (default, Dec 21 2022, 04:00:16)
[Clang 14.0.3 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "/home/charles/program/python/test_nvidia/.venv/lib/python3.8/site-packages/torch/__init__.py", line 168, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/charles/.rye/py/cpython@3.8.16/install/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcurand.so.10: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/charles/program/python/test_nvidia/.venv/lib/python3.8/site-packages/torch/__init__.py", line 228, in <module>
    _load_global_deps()
  File "/home/charles/program/python/test_nvidia/.venv/lib/python3.8/site-packages/torch/__init__.py", line 189, in _load_global_deps
    _preload_cuda_deps(lib_folder, lib_name)
  File "/home/charles/program/python/test_nvidia/.venv/lib/python3.8/site-packages/torch/__init__.py", line 154, in _preload_cuda_deps
    raise ValueError(f"{lib_name} not found in the system path {sys.path}")
ValueError: libcublas.so.*[0-9] not found in the system path ['', '/home/charles/.rye/py/cpython@3.8.16/install/lib/python38.zip', '/home/charles/.rye/py/cpython@3.8.16/install/lib/python3.8', '/home/charles/.rye/py/cpython@3.8.16/install/lib/python3.8/lib-dynload', '/home/charles/program/python/test_nvidia/.venv/lib/python3.8/site-packages', '/home/charles/program/python/test_nvidia/src']