rye sync fails to find nvidia-cublas-cu12 even though uv pip install works

rachtsingh commented 8 months ago

Steps to Reproduce

uv can find this package fine:

~/proj/uv-test ❯ which uv
/home/singhrac/.cargo/bin/uv
~/proj/uv-test ❯ uv venv
Using Python 3.12.1 interpreter at: /home/singhrac/.rye/py/cpython@3.12.1/install/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
~/proj/uv-test ❯ uv pip install nvidia-cublas-cu12==12.4.2.65
Resolved 1 package in 71ms
Downloaded 1 package in 17.49s
Installed 1 package in 10ms
 + nvidia-cublas-cu12==12.4.2.65

Trying the same with rye works (when uv is turned off):

~/proj ❯ rye init rye-test
success: Initialized project in /home/singhrac/proj/rye-test
  Run `rye sync` to get started
[project]
~/proj ❯ cd rye-test
~/proj ❯ vim pyproject.toml
~/proj/rye-test main ?6 ❯ cat pyproject.toml
[project]
name = "rye-test"
version = "0.1.0"
description = "Add your description here"
authors = [
    { name = "Rachit Singh", email = "rachitsingh@outlook.com" }
]
dependencies = [
    "jax[cuda12_pip]>=0.4.25",
]
readme = "README.md"
requires-python = ">= 3.8"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.rye]
managed = true
dev-dependencies = []

[tool.hatch.metadata]
allow-direct-references = true

[tool.hatch.build.targets.wheel]
packages = ["src/rye_test"]

[[tool.rye.sources]]
name = "google-jax"
url = "https://storage.googleapis.com/jax-releases/jax_cuda_releases.html"
type = "find-links"
~/proj/rye-test main ?6 ❯ rye config --set-bool behavior.use-uv=false
~/proj/rye-test main ?6 ❯ rye add "jax[cuda12_pip]"
Added jax[cuda12_pip]>=0.4.25 as regular dependency
~/proj/rye-test main ?6 ❯ rye sync
...
Successfully installed jax-0.4.25 jaxlib-0.4.25+cuda12.cudnn89 ml-dtypes-0.3.2 numpy-1.26.4 nvidia-cublas-cu12-12.4.2.65 nvidia-cuda-cupti-cu12-12.4.99 nvidia-cuda-nvcc-cu12-12.4.99 nvidia-cuda-nvrtc-cu12-12.4.99 nvidia-cuda-runtime-cu12-12.4.99 nvidia-cudnn-cu12-8.9.7.29 nvidia-cufft-cu12-11.2.0.44 nvidia-cusolver-cu12-11.6.0.99 nvidia-cusparse-cu12-12.3.0.142 nvidia-nccl-cu12-2.20.5 nvidia-nvjitlink-cu12-12.4.99 opt-einsum-3.3.0 rye-test-0.1.0 scipy-1.12.0
Done!

but when we set rye to use uv, it fails to resolve:

~/proj/rye-test main ?7 ❯ cp pyproject.toml ../test.toml                                                                                                4m 10s
~/proj/rye-test main ?7 ❯ cd ..
~/proj ❯ rm -rf rye-test
~/proj ❯ rye init rye-test
success: Initialized project in /home/singhrac/proj/rye-test
  Run `rye sync` to get started
~/proj ❯ cp test.toml rye-test/pyproject.toml
~/proj ❯ cd rye-test
~/proj/rye-test main ?5 ❯ rye config --set-bool behavior.use-uv=true
~/proj/rye-test main ?5 ❯ rye sync
Initializing new virtualenv in /home/singhrac/proj/rye-test/.venv
Python version: cpython@3.12.1
Generating production lockfile: /home/singhrac/proj/rye-test/requirements.lock
  × No solution found when resolving dependencies:
  ╰─▶ Because only nvidia-cublas-cu12<12.3.4.1 is available and jax[cuda12-pip]==0.4.25 depends on nvidia-cublas-cu12>=12.3.4.1, we can conclude that
      jax[cuda12-pip]==0.4.25 cannot be used.
      And because only jax[cuda12-pip]<=0.4.25 is available and you require jax[cuda12-pip]>=0.4.25, we can conclude that the requirements are unsatisfiable.
error: could not write production lockfile for project

Caused by:
    failed to generate lockfile

Expected Result

The middle outcome, where rye sync succeeds.

Actual Result

Last outcome above, i.e. "failed to generate lockfile".

Version Info

~/proj/rye-test main ?5 ❯ rye --version
rye 0.29.0
commit: 0.29.0 (42b179f36 2024-03-11)
platform: linux (x86_64)
self-python: cpython@3.12.1
symlink support: true
uv enabled: true

and separately

~/proj/rye-test main ?5 ❯ uv version
uv 0.1.18

Maybe it's because rye's version of uv is a few weeks old? I think uv is now on 0.1.18 and rye uses 0.1.7? Oh, that seems to be a typo in CHANGELOG.md, since previously it was on 0.1.15. I'll submit a PR for that.

Stacktrace

No response

rachtsingh commented 8 months ago

By the way, I ran into this when trying to check if https://github.com/astral-sh/rye/issues/745 is fixed - I think this is essentially the remaining issue.

bluss commented 8 months ago

Your pyproject has requires-python >= 3.8, could you try to change that and try again?

I see uv failing that way too

echo 'jax[cuda12_pip]>=0.4.25' | uv pip compile -p 3.8  -  -f 'https://storage.googleapis.com/jax-releases/jax_cuda_releases.html'
warning: The requested Python version 3.8 is not available; 3.10.12 will be used to build dependencies instead.
  × No solution found when resolving dependencies

while it succeeds for python 3.9 or later.

With that said, I'm not sure that's what actually happens in rye.. is it using requires python that way? should it?

FWIW, I have the same version of rye and don't have this problem with your pyproject.toml.

rachtsingh commented 8 months ago

Same issue:

~/proj/rye-test main ?5 ❯ cat pyproject.toml                                                                                                                                    8s
[project]
name = "rye-test"
version = "0.1.0"
description = "Add your description here"
authors = [
    { name = "Rachit Singh", email = "rachitsingh@outlook.com" }
]
dependencies = [
    "jax[cuda12_pip]>=0.4.25",
]
readme = "README.md"
requires-python = ">= 3.12"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.rye]
managed = true
dev-dependencies = []

[tool.hatch.metadata]
allow-direct-references = true

[tool.hatch.build.targets.wheel]
packages = ["src/rye_test"]

[[tool.rye.sources]]
name = "google-jax"
url = "https://storage.googleapis.com/jax-releases/jax_cuda_releases.html"
type = "find-links"
~/proj/rye-test main ?5 ❯ rye sync
Initializing new virtualenv in /home/singhrac/proj/rye-test/.venv
Python version: cpython@3.12.1
Generating production lockfile: /home/singhrac/proj/rye-test/requirements.lock
  × No solution found when resolving dependencies:
  ╰─▶ Because only nvidia-cublas-cu12<12.3.4.1 is available and jax[cuda12-pip]==0.4.25 depends on nvidia-cublas-cu12>=12.3.4.1, we can conclude that jax[cuda12-pip]==0.4.25
      cannot be used.
      And because only jax[cuda12-pip]<=0.4.25 is available and you require jax[cuda12-pip]>=0.4.25, we can conclude that the requirements are unsatisfiable.
error: could not write production lockfile for project

Caused by:
    failed to generate lockfile

You're not able to reproduce the issue? Have you checked that rye is configured to use uv (see my note above about non-uv usage succeeding), and that the pip cache is empty?

bluss commented 8 months ago

Not sure how to ensure pip cache is empty, but it still succeeds in the resolution like this -- using your pyproject.toml.

XDG_CACHE_HOME=$PWD UV_NO_CACHE=true rye lock
Initializing new virtualenv in /home/user/ryes/rye-test/.venv
Python version: cpython@3.12.2
Generating production lockfile: /home/user/ryes/rye-test/requirements.lock
Generating dev lockfile: /home/user/ryes/rye-test/requirements-dev.lock
Done!

rye 0.29.0
commit: 0.29.0 (42b179f36 2024-03-11)
platform: linux (x86_64)
self-python: cpython@3.12.2
symlink support: true
uv enabled: true

requirements.lock

``` # generated by rye # use `rye lock` or `rye sync` to update this lockfile # # last locked with the following flags: # pre: false # features: [] # all-features: false # with-sources: false -e file:. jax==0.4.25 # via rye-test jaxlib==0.4.25+cuda12.cudnn89 # via jax ml-dtypes==0.3.2 # via jax # via jaxlib numpy==1.26.4 # via jax # via jaxlib # via ml-dtypes # via opt-einsum # via scipy nvidia-cublas-cu12==12.4.2.65 # via jax # via nvidia-cudnn-cu12 # via nvidia-cusolver-cu12 nvidia-cuda-cupti-cu12==12.4.99 # via jax nvidia-cuda-nvcc-cu12==12.4.99 # via jax nvidia-cuda-nvrtc-cu12==12.4.99 # via nvidia-cudnn-cu12 nvidia-cuda-runtime-cu12==12.4.99 # via jax nvidia-cudnn-cu12==8.9.7.29 # via jax nvidia-cufft-cu12==11.2.0.44 # via jax nvidia-cusolver-cu12==11.6.0.99 # via jax nvidia-cusparse-cu12==12.3.0.142 # via jax # via nvidia-cusolver-cu12 nvidia-nccl-cu12==2.20.5 # via jax nvidia-nvjitlink-cu12==12.4.99 # via jax # via nvidia-cusolver-cu12 # via nvidia-cusparse-cu12 opt-einsum==3.3.0 # via jax scipy==1.12.0 # via jax # via jaxlib ```

rachtsingh commented 8 months ago

I'm at a loss, I have no idea why this would succeed for you but not for me. Maybe I need a fresh install of rye? I've been using rye self update.

God-damnit-all commented 7 months ago

I'm also having this issue when trying to install tensorflow[and-cuda]==2.16.1, I get the error:

error: Failed to run uv compile   × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of nvidia-cublas-cu12==12.3.4.1 and
      you require nvidia-cublas-cu12==12.3.4.1, we can conclude that the
      requirements are unsatisfiable.
. uv exited with status: exit code: 1

Even though it's right here: https://pypi.org/project/nvidia-cublas-cu12/12.3.4.1/

ash-sony commented 4 months ago

I'm also having this issue when trying to install tensorflow[and-cuda]==2.16.1, I get the error:
error: Failed to run uv compile   × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of nvidia-cublas-cu12==12.3.4.1 and
      you require nvidia-cublas-cu12==12.3.4.1, we can conclude that the
      requirements are unsatisfiable.
. uv exited with status: exit code: 1
Even though it's right here: https://pypi.org/project/nvidia-cublas-cu12/12.3.4.1/

nvidia cublas doesnt support python 3.12, unfortunately. Pip's error messaging is confusing because it raises that error when there's no version available matching your current Python

astral-sh / rye