search

astral-sh / uv

An extremely fast Python package and project manager, written in Rust.
https://docs.astral.sh/uv
Apache License 2.0
28.06k stars 806 forks source link

Dependency resolution failed in offline mode #9277

Open Winand opened 1 week ago

Winand commented 1 week ago

I'm trying to install a project in offline environment on CentOS 7. Here's a project with numpy and pytest dependencies.

First I download all the dependencies for installation on CentOS 7 using uv export and pip download. See the script.

On the target system I've put offline=true and no-index=true in uv global settings file. Then I run uv sync and get an error _"no versions of colorama{sysplatform == 'win32'}". But I'm trying to install on Linux not win32.

[/tmp/uv-offline-installation-bug]$ uv sync -f packages -v
DEBUG uv 0.5.3
DEBUG Found project root: `/tmp/uv-offline-installation-bug`
DEBUG No workspace root found, using project root
DEBUG Reading Python requests from version file at `/tmp/uv-offline-installation-bug/.python-version`
DEBUG Using Python request `3.12` from version file at `.python-version`
DEBUG The virtual environment's Python version satisfies `3.12`
DEBUG Using request timeout of 30s
DEBUG Found 5 packages in `--find-links` entry: file:///tmp/uv-offline-installation-bug/packages
DEBUG Found static `pyproject.toml` for: uv-offline-installation-bug @ file:///tmp/uv-offline-installation-bug
DEBUG No workspace root found, using project root
DEBUG Ignoring existing lockfile due to missing remote index: `numpy` `2.1.3` from `https://pypi.org/simple`
DEBUG Solving with installed Python version: 3.12.3
DEBUG Solving with target Python version: >=3.12
DEBUG Adding direct dependency: uv-offline-installation-bug*
DEBUG Searching for a compatible version of uv-offline-installation-bug @ file:///tmp/uv-offline-installation-bug (*)
DEBUG Adding transitive dependency for uv-offline-installation-bug==0.1.0: numpy>=2.1.3
DEBUG Adding transitive dependency for uv-offline-installation-bug==0.1.0: uv-offline-installation-bug:dev==0.1.0
DEBUG Searching for a compatible version of uv-offline-installation-bug @ file:///tmp/uv-offline-installation-bug (==0.1.0)
DEBUG Adding transitive dependency for uv-offline-installation-bug==0.1.0: uv-offline-installation-bug==0.1.0
DEBUG Adding transitive dependency for uv-offline-installation-bug==0.1.0: uv-offline-installation-bug:dev==0.1.0
DEBUG Searching for a compatible version of uv-offline-installation-bug @ file:///tmp/uv-offline-installation-bug (==0.1.0)
DEBUG Adding transitive dependency for uv-offline-installation-bug==0.1.0: pytest>=8.3.3
DEBUG Searching for a compatible version of numpy (>=2.1.3)
DEBUG Selecting: numpy==2.1.3 [preference] (numpy-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl)
DEBUG Searching for a compatible version of pytest (>=8.3.3)
DEBUG Selecting: pytest==8.3.3 [preference] (pytest-8.3.3-py3-none-any.whl)
DEBUG Adding transitive dependency for pytest==8.3.3: colorama{sys_platform == 'win32'}*
DEBUG Adding transitive dependency for pytest==8.3.3: iniconfig*
DEBUG Adding transitive dependency for pytest==8.3.3: packaging*
DEBUG Adding transitive dependency for pytest==8.3.3: pluggy>=1.5, <2
DEBUG Searching for a compatible version of colorama{sys_platform == 'win32'} (*)
DEBUG No compatible version found for: colorama{sys_platform == 'win32'}
DEBUG Searching for a compatible version of numpy (>=2.1.3)
DEBUG Selecting: numpy==2.1.3 [preference] (numpy-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl)
DEBUG Searching for a compatible version of pytest (>8.3.3)
DEBUG No compatible version found for: pytest
DEBUG Searching for a compatible version of uv-offline-installation-bug @ file:///tmp/uv-offline-installation-bug (<0.1.0 | >0.1.0)
DEBUG No compatible version found for: uv-offline-installation-bug
  × No solution found when resolving dependencies:
  ╰─▶ Because there are no versions of colorama{sys_platform == 'win32'} and pytest==8.3.3 depends on colorama{sys_platform
      == 'win32'}, we can conclude that pytest==8.3.3 cannot be used.
      And because only pytest==8.3.3 is available, we can conclude that all versions of pytest cannot be used.
      And because uv-offline-installation-bug:dev depends on pytest and your project depends on
      uv-offline-installation-bug:dev, we can conclude that your project's requirements are unsatisfiable.
konstin commented 1 week ago

You can add --frozen to avoid uv doing the resolution, this will make your command pass (e.g. uv sync -f packages -v --no-index --offline --frozen). I'm a bit surprised uv is trying to re-resolve here, maybe due to the new find-links

Winand commented 1 week ago

Unfortunately after adding --frozen I encountered an error as if --no-index had not been specified:

[/tmp/uv-offline-installation-bug]$ uv sync -f packages -v --no-index --offline --frozen
DEBUG uv 0.5.3
DEBUG Found project root: `/tmp/uv-offline-installation-bug`
DEBUG No workspace root found, using project root
DEBUG Reading Python requests from version file at `/tmp/uv-offline-installation-bug/.python-version`
DEBUG Using Python request `3.12` from version file at `.python-version`
DEBUG The virtual environment's Python version satisfies `3.12`
DEBUG Using request timeout of 30s
DEBUG Found 5 packages in `--find-links` entry: file:///tmp/uv-offline-installation-bug/packages
DEBUG Identified uncached distribution: pytest==8.3.3
DEBUG Identified uncached distribution: numpy==2.1.3
DEBUG Identified uncached distribution: iniconfig==2.0.0
DEBUG Identified uncached distribution: packaging==24.2
DEBUG Identified uncached distribution: pluggy==1.5.0
DEBUG No cache entry for: https://files.pythonhosted.org/packages/9e/3e/3757f304c704f2f0294a6b8340fcf2be244038be07da4cccf390fa678a9f/numpy-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  × Failed to download `numpy==2.1.3`
  ╰─▶ Network connectivity is disabled, but the requested data wasn't found in the cache for:
      `https://files.pythonhosted.org/packages/9e/3e/3757f304c704f2f0294a6b8340fcf2be244038be07da4cccf390fa678a9f/numpy-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`
  help: `numpy` (v2.1.3) was included because `uv-offline-installation-bug` (v0.1.0) depends on `numpy`

For comparison:

[/tmp/uv-offline-installation-bug]$ uv sync -f packages -v --offline --no-config
DEBUG uv 0.5.3
DEBUG Found project root: `/tmp/uv-offline-installation-bug`
DEBUG No workspace root found, using project root
DEBUG Ignoring Python version file at `.python-version` due to `--no-config`
DEBUG Using Python request `>=3.12` from `requires-python` metadata
DEBUG The virtual environment's Python version satisfies `>=3.12`
DEBUG Using request timeout of 30s
DEBUG Found 5 packages in `--find-links` entry: file:///tmp/uv-offline-installation-bug/packages
DEBUG Found static `pyproject.toml` for: uv-offline-installation-bug @ file:///tmp/uv-offline-installation-bug
DEBUG No workspace root found, using project root
DEBUG Existing `uv.lock` satisfies workspace requirements
Resolved 7 packages in 3ms
DEBUG Using request timeout of 30s
DEBUG Found 5 packages in `--find-links` entry: file:///tmp/uv-offline-installation-bug/packages
DEBUG Identified uncached distribution: pytest==8.3.3
DEBUG Identified uncached distribution: numpy==2.1.3
DEBUG Identified uncached distribution: iniconfig==2.0.0
DEBUG Identified uncached distribution: packaging==24.2
DEBUG Identified uncached distribution: pluggy==1.5.0
DEBUG No cache entry for: https://files.pythonhosted.org/packages/9e/3e/3757f304c704f2f0294a6b8340fcf2be244038be07da4cccf390fa678a9f/numpy-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  × Failed to download `numpy==2.1.3`
  ╰─▶ Network connectivity is disabled, but the requested data wasn't found in the cache for:
      `https://files.pythonhosted.org/packages/9e/3e/3757f304c704f2f0294a6b8340fcf2be244038be07da4cccf390fa678a9f/numpy-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`
  help: `numpy` (v2.1.3) was included because `uv-offline-installation-bug` (v0.1.0) depends on `numpy`
konstin commented 1 week ago

The problem is that uv wants to sync from the lockfile, but the lockfile has the packages from pypi locked, and that fails in offline mode if the cache isn't warm (as it was in my case). With -f packages, uv will do a different resolution, locking the packages directory, you can see that in uv.lock after doing uv sync -f packages -v --no-index in online mode. Currently, uv has no support for replacing files from a registry with local files, it only works from cache.

Winand commented 1 week ago

~The only solution I've found is to set the folder packages as a default index. Lock file is updated but i can backup and restore it afterwards, so I don't have unstaged changes in repo.~

Log

``` [/tmp/uv-offline-installation-bug]$ uv sync --default-index packages --offline --no-config -v DEBUG uv 0.5.3 DEBUG Found project root: `/tmp/uv-offline-installation-bug` DEBUG No workspace root found, using project root DEBUG Ignoring Python version file at `.python-version` due to `--no-config` DEBUG Using Python request `>=3.12` from `requires-python` metadata <...> Using CPython 3.12.3 interpreter at: /opt/apps/user/.local/bin/python3.12 Creating virtual environment at: .venv DEBUG Using request timeout of 30s DEBUG Found static `pyproject.toml` for: uv-offline-installation-bug @ file:///tmp/uv-offline-installation-bug DEBUG No workspace root found, using project root DEBUG Existing `uv.lock` satisfies workspace requirements Resolved 7 packages in 1ms DEBUG Using request timeout of 30s DEBUG Requirement already cached: pytest==8.3.3 DEBUG Requirement already cached: numpy==2.1.3 DEBUG Requirement already cached: iniconfig==2.0.0 DEBUG Requirement already cached: packaging==24.2 DEBUG Requirement already cached: pluggy==1.5.0 <...> warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance. If the cache and target directories are on different filesystems, hardlinking may not be supported. If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning. Installed 5 packages in 747ms + iniconfig==2.0.0 + numpy==2.1.3 + packaging==24.2 + pluggy==1.5.0 + pytest==8.3.3 ```

notatallshaw-gts commented 1 week ago

The problem is that uv wants to sync from the lockfile, but the lockfile has the packages from pypi locked, and that fails in offline mode if the cache isn't warm (as it was in my case).

Would it make sense (or at improve user experience) that when a user provides --offline to assume any cache timeout is infinity?

Winand commented 1 week ago

The only solution I've found is to set the folder packages as a default index. Lock file is updated but i can backup and restore it afterwards, so I don't have unstaged changes in repo.

hmm, it works only with "warm" cache. But I don't remember after which command the cache was updated if I don't have internet access. upd: i think the cache was updated after I added packages from pip download results on Windows so uv could find colorama.

Winand commented 1 week ago

Ok, this solution works.

~/.config/uv/uv.toml:

offline = true
no-index = true

Export to requirements file and install using pip interface:

uv venv
uv export --no-hashes --frozen -o /tmp/requirements.txt
uv pip install -f packages -r /tmp/requirements.txt
rm /tmp/requirements.txt

The only issue is that I need the project to be installed as a package. So I added it to [tool.hatch.build.targets.wheel] block and now I have to declare hatchling and editables as dev dependencies to download them using pip download.