Support Python version upper bound in `uv pip compile --universal`

Lordshinjo commented 3 weeks ago

Using uv 0.2.23, the following fails to resolve:

echo 'persistent>=6.0' | uv pip compile --universal --python-version 3.9 -
  × No solution found when resolving dependencies:
  ╰─▶ Because only cffi{platform_python_implementation == 'CPython' and python_version >= '3.13a0'}<1.17.0rc1 is available and persistent==6.0 depends on cffi{platform_python_implementation == 'CPython'
      and python_version >= '3.13a0'}>=1.17.0rc1, we can conclude that persistent==6.0 cannot be used.
      And because only persistent<=6.0 is available and you require persistent>=6.0, we can conclude that the requirements are unsatisfiable.

The issue here is that persistent==6.0 depends on the following:

cffi ; platform_python_implementation == 'CPython'
cffi >= 1.17.0rc1 ; platform_python_implementation == 'CPython' and python_version >= '3.13a0'

The 1st one is fine and resolves to cffi==1.16.0 without --universal, however the 2nd one is a dependency on a prerelease-only package version range for a prerelease-only Python version, and as such fails unless we enable all prerelease versions.

Being able to specify an upper bound (e.g. --max-python-version 3.12, though an exclusive bound might be better) would allow for a successful resolution.

I was also wondering whether this issue is a bug since this concerns a prerelease Python version that should probably be ignored, but:

AFAIK there's no easy way to know which Python versions are available like package versions.
The generated requirements may last until the Python version is fully released, so it might make sense to explicit which Python versions are supported.

BurntSushi commented 3 weeks ago

Hmm, #4732 might be related here. In particular, the marker expressions are overlapping, so I think the universal resolver is probably not forking in this case. And indeed, trying it out locally, I can see that no forks occur. But, because there are no forks, it seems to me like cffi and cffi >= 1.17.0rc1 should combine just fine since 1.17.0rc1 has a Requires-Python of >=3.8. Indeed, if I do uv pip compile on just the cffi dependencies, then it works:

[andrew@duff uv]$ cat /tmp/req.in
cffi ; platform_python_implementation == 'CPython'
cffi >= 1.17.0rc1 ; platform_python_implementation == 'CPython' and python_version >= '3.13a0'
[andrew@duff uv]$ uv pip compile --universal --python-version 3.9 /tmp/req.in
Resolved 2 packages in 3ms
# This file was autogenerated by uv via the following command:
#    uv pip compile --universal --python-version 3.9 /tmp/req.in
cffi==1.17.0rc1 ; platform_python_implementation == 'CPython'
    # via -r /tmp/req.in
pycparser==2.22 ; platform_python_implementation == 'CPython'
    # via cffi

OK, so looking at the metadata for persistent 6.0, I see:

Requires-Dist: zope.deferredimport
Requires-Dist: zope.interface
Requires-Dist: cffi ; platform_python_implementation == "CPython"
Requires-Dist: cffi >=1.17.0rc1 ; platform_python_implementation == "CPython" and python_version >= "3.13a0"
Provides-Extra: docs
Requires-Dist: Sphinx ; extra == 'docs'
Requires-Dist: repoze.sphinx.autointerface ; extra == 'docs'
Requires-Dist: sphinx-rtd-theme ; extra == 'docs'
Provides-Extra: test
Requires-Dist: zope.testrunner ; extra == 'test'
Requires-Dist: manuel ; extra == 'test'
Provides-Extra: testing

Converting that to requirements.in, I get:

zope.deferredimport
zope.interface
cffi ; platform_python_implementation == "CPython"
cffi >=1.17.0rc1 ; platform_python_implementation == "CPython" and python_version >= "3.13a0"

And running uv pip compile with that seems to work fine?

[andrew@duff uv]$ uv pip compile --universal --python-version 3.9 /tmp/req-persistent.in
Resolved 6 packages in 4ms
# This file was autogenerated by uv via the following command:
#    uv pip compile --universal --python-version 3.9 /tmp/req-persistent.in
cffi==1.17.0rc1 ; platform_python_implementation == 'CPython'
    # via -r /tmp/req-persistent.in
pycparser==2.22 ; platform_python_implementation == 'CPython'
    # via cffi
setuptools==70.3.0
    # via
    #   zope-deferredimport
    #   zope-interface
    #   zope-proxy
zope-deferredimport==5.0
    # via -r /tmp/req-persistent.in
zope-interface==6.4.post2
    # via
    #   -r /tmp/req-persistent.in
    #   zope-proxy
zope-proxy==5.2
    # via zope-deferredimport

But indeed, just persistent >= 6.0 fails:

[andrew@duff uv]$ echo 'persistent>=6.0' | uv pip compile --universal --python-version 3.9 - -v
DEBUG uv 0.2.23
DEBUG Starting Python discovery for Python 3.9
DEBUG Looking for exact match for request Python 3.9
DEBUG Searching for Python 3.9 in system path
DEBUG Found cpython 3.9.18 at `/home/andrew/.pyenv/shims/python3.9` (search path)
DEBUG Using Python 3.9.18 interpreter at /home/andrew/.pyenv/versions/3.9.18/bin/python3.9 for builds
DEBUG Using request timeout of 30s
DEBUG Solving with installed Python version: 3.9.18
DEBUG Solving with target Python version: >=3.9
DEBUG Adding direct dependency: persistent>=6.0
DEBUG Found fresh response for: https://pypi.org/simple/persistent/
DEBUG Searching for a compatible version of persistent (>=6.0)
DEBUG Selecting: persistent==6.0 (persistent-6.0-cp310-cp310-macosx_10_9_x86_64.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/b3/06/2e26cf460e12322654280babd7cbcd5d8bccf123cc73db818b523ba32fe5/persistent-6.0-cp310-cp310-macosx_10_9_x86_64.whl.metadata
DEBUG Adding transitive dependency for persistent==6.0: zope-deferredimport*
DEBUG Adding transitive dependency for persistent==6.0: zope-interface*
DEBUG Adding transitive dependency for persistent==6.0: cffi{platform_python_implementation == 'CPython'}*
DEBUG Adding transitive dependency for persistent==6.0: cffi{platform_python_implementation == 'CPython' and python_version >= '3.13a0'}>=1.17.0rc1
DEBUG Found fresh response for: https://pypi.org/simple/zope-deferredimport/
DEBUG Searching for a compatible version of zope-deferredimport (*)
DEBUG Selecting: zope-deferredimport==5.0 (zope.deferredimport-5.0-py3-none-any.whl)
DEBUG Found fresh response for: https://pypi.org/simple/zope-interface/
DEBUG Found fresh response for: https://pypi.org/simple/cffi/
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/69/f7/b5e232857f4d511b2628697bf2e48fca55ea4ed75b1432efd1bf024fbd12/zope.deferredimport-5.0-py3-none-any.whl.metadata
DEBUG Adding transitive dependency for zope-deferredimport==5.0: setuptools*
DEBUG Adding transitive dependency for zope-deferredimport==5.0: zope-proxy*
DEBUG Searching for a compatible version of zope-interface (*)
DEBUG Selecting: zope-interface==6.4.post2 (zope.interface-6.4.post2-cp310-cp310-macosx_10_9_x86_64.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/55/3c/a508767ab863573ee5ccca1a57389164017a55cfb61b1cb357882e9ed553/zope.interface-6.4.post2-cp310-cp310-macosx_10_9_x86_64.whl.metadata
DEBUG Adding transitive dependency for zope-interface==6.4.post2: setuptools*
DEBUG Searching for a compatible version of cffi{platform_python_implementation == 'CPython'} (*)
DEBUG Selecting: cffi==1.16.0 (cffi-1.16.0-cp310-cp310-macosx_10_9_x86_64.whl)
DEBUG Adding transitive dependency for cffi==1.16.0: cffi==1.16.0
DEBUG Adding transitive dependency for cffi==1.16.0: cffi{platform_python_implementation == 'CPython'}==1.16.0
DEBUG Searching for a compatible version of cffi{platform_python_implementation == 'CPython'} (==1.16.0)
DEBUG Selecting: cffi==1.16.0 (cffi-1.16.0-cp310-cp310-macosx_10_9_x86_64.whl)
DEBUG Found fresh response for: https://pypi.org/simple/zope-proxy/
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/aa/aa/1c43e48a6f361d1529f9e4602d6992659a0107b5f21cae567e2eddcf8d66/cffi-1.16.0-cp310-cp310-macosx_10_9_x86_64.whl.metadata
DEBUG Adding transitive dependency for cffi==1.16.0: pycparser*
DEBUG Searching for a compatible version of cffi (==1.16.0)
DEBUG Found stale response for: https://pypi.org/simple/setuptools/
DEBUG Selecting: cffi==1.16.0 (cffi-1.16.0-cp310-cp310-macosx_10_9_x86_64.whl)
DEBUG Sending revalidation request for: https://pypi.org/simple/setuptools/
DEBUG Adding transitive dependency for cffi==1.16.0: pycparser*
DEBUG Searching for a compatible version of cffi{platform_python_implementation == 'CPython' and python_version >= '3.13a0'} (>=1.17.0rc1)
DEBUG No compatible version found for: cffi{platform_python_implementation == 'CPython' and python_version >= '3.13a0'}
DEBUG Searching for a compatible version of persistent (>6.0)
DEBUG No compatible version found for: persistent
  x No solution found when resolving dependencies:
  `-> Because only cffi{platform_python_implementation == 'CPython' and python_version >= '3.13a0'}<1.17.0rc1 is available and persistent==6.0 depends on cffi{platform_python_implementation == 'CPython' and python_version >=
      '3.13a0'}>=1.17.0rc1, we can conclude that persistent==6.0 cannot be used.
      And because only persistent<=6.0 is available and you require persistent>=6.0, we can conclude that the requirements are unsatisfiable.

Ah okay, now I think I get it. This is the key piece I was missing:

and as such fails unless we enable all prerelease versions.

Right, so even though cffi==1.17.0rc1 satisfies both dependency specifications, the intent here is that cffi ; platform_python_implementation == "CPython" should select a non-pre-release version. But that cffi >=1.17.0rc1 ; platform_python_implementation == "CPython" and python_version >= "3.13a0" should select a pre-release version since it's the only one available given the constraints. So the correct universal lock file here should have two different versions of cffi in it.

I do think that #4732 should be able to fix this. Namely, if we modify the universal resolver to fork in this case, then I think it should be possible to do the right thing here. With that said, I'm not sure we've fully tackled the interaction points between pre-releases and universal resolution yet.

konstin commented 3 weeks ago

In addition to #4732 and the great explanation by @BurntSushi, i think we also need https://github.com/astral-sh/uv/issues/4579 to solve.

astral-sh / uv

Support Python version upper bound in `uv pip compile --universal` #4959