astral-sh / uv

An extremely fast Python package and project manager, written in Rust.
https://docs.astral.sh/uv
Apache License 2.0
24.69k stars 712 forks source link

uv pip compile --universal fails when dependency's python-requires is incompatible with -p, even with markers #6730

Closed alex closed 1 month ago

alex commented 2 months ago
~ ❯❯❯ uv version
uv 0.3.5 (Homebrew 2024-08-27)

When you use a marker to express a dependency, uv pip compile --universal "forwards" that to the output. For example:

~ ❯❯❯ echo 'argcomplete; python_version >= "3.8"' | uv pip compile --universal -p 3.7 -
warning: The requested Python version 3.7 is not available; 3.12.5 will be used to build dependencies instead.
Resolved 4 packages in 3ms
# This file was autogenerated by uv via the following command:
#    uv pip compile --universal -p 3.7 -
argcomplete==3.1.2 ; python_full_version >= '3.8'
importlib-metadata==6.7.0 ; python_version < '0'
    # via argcomplete
typing-extensions==4.7.1 ; python_version < '0'
    # via importlib-metadata
zipp==3.15.0 ; python_version < '0'
    # via importlib-metadata

When Python 3.7 is actually used, no dependencies will be installed.

However, when this same marker is used for a dependency whose python-requires mean there is no resolution for the lower version, this fails:

~ ❯❯❯ echo 'check-sdist; python_version >= "3.8"' | uv pip compile --universal -p 3.7 -
warning: The requested Python version 3.7 is not available; 3.12.5 will be used to build dependencies instead.
  × No solution found when resolving dependencies:
  ╰─▶ Because only check-sdist{python_full_version >= '3.8'}<=0.1.3 is available and the requested Python version (>=3.7)
      does not satisfy Python>=3.8, we can conclude that all versions of check-sdist{python_full_version >= '3.8'} are
      incompatible.
      And because you require check-sdist{python_full_version >= '3.8'}, we can conclude that your requirements are
      unsatisfiable.

The fact that check-sdist only supports Python 3.8+ should be ok, because the dependency has the ; python_version >= "3.8" marker. But instead uv pip compile rejects it because it's not compatible with -p.

charliermarsh commented 2 months ago

Ah yeah. You can solve this with the environments field, I think? By setting environments = ["python_version >= '3.8'"]. I think it is similar to https://github.com/astral-sh/uv/issues/4668

alex commented 2 months ago

Hmm, this may in fact be a duplicate of that issue.

I'm not sure environments does what I want here -- I want to have a lockfile that installs correctly on Python 3.7, but does not include this dependency.

alex commented 2 months ago

Ah, environments = ["python_version < '3.8'", "python_version >= '3.8'"] causes it to work though.

charliermarsh commented 2 months ago

I think (or hope) that environments = ["python_version < '3.8'", "python_version >= '3.8'"] would work though. That should work on any.

charliermarsh commented 2 months ago

Yeah. The reason this happens is related to implementation details of the resolver (that we're considering changing). We only "fork" the resolver if we see multiple requirements for the same package. Once we fork, we (e.g.) do the right thing with respect to narrowing the supported Python range. But if we only see argcomplete; python_version >= "3.8", we don't fork (at least, not right now -- we have a PR open that does do this).

alex commented 2 months ago

This may be a seperate issue, or not an issue at all, but it also seems that markers don't influence version resolution? Compare:

~ ❯❯❯ echo 'argcomplete; python_version >= "3.8"' | uv pip compile --universal -p 3.7 -
warning: The requested Python version 3.7 is not available; 3.12.5 will be used to build dependencies instead.
Resolved 4 packages in 3ms
# This file was autogenerated by uv via the following command:
#    uv pip compile --universal -p 3.7 -
argcomplete==3.1.2 ; python_full_version >= '3.8'
importlib-metadata==6.7.0 ; python_version < '0'
    # via argcomplete
typing-extensions==4.7.1 ; python_version < '0'
    # via importlib-metadata
zipp==3.15.0 ; python_version < '0'
    # via importlib-metadata
~ ❯❯❯ echo 'argcomplete; python_version >= "3.8"' | uv pip compile --universal -p 3.8 -
warning: The requested Python version 3.8 is not available; 3.12.5 will be used to build dependencies instead.
Resolved 1 package in 1ms
# This file was autogenerated by uv via the following command:
#    uv pip compile --universal -p 3.8 -
argcomplete==3.5.0
charliermarsh commented 2 months ago

Can you explain that comment a bit more? Sorry, trying to follow. Do you mean the -p 3.7 vs. -p 3.8?

alex commented 2 months ago

Sorry, I should have written more words.

In both of these invocations, argcomplete is only for Python 3.8. In one version this is done with -p 3.8 and in the other it's done with ; python_version >= "3.8". But these resolve to different argcomplete versions.

charliermarsh commented 2 months ago

Ohhh yes, I didn't notice that. I think that's because in the non-forking case (without environments), we're still trying to select a version that works on Python 3.7.

environments is kind of a workaround for this right now; the resolver should actually do the right thing here, and select the maximum compatible version for Python 3.8 or later.

alex commented 2 months ago

Should this be a separate issue, or is this the right place?

alex commented 1 month ago

I seem to have found another variant on this -- when you have additional environments:

(tempenv-70b9155915899) ~/.v/tempenv-70b9155915899 ❯❯❯ cat uv.toml 
environments = ["python_version >= '3.10'", "python_version >= '3.8' and python_version < '3.10'", "python_version < '3.8'"]

(tempenv-70b9155915899) ~/.v/tempenv-70b9155915899 ❯❯❯ echo 'check-sdist; python_version >= "3.8"' | uv pip compile --universal -p 3.7 -
warning: The requested Python version 3.7 is not available; 3.12.5 will be used to build dependencies instead.
  × No solution found when resolving dependencies for split (python_full_version >= '3.8' and python_full_version <
  │ '3.10'):
  ╰─▶ Because only check-sdist{python_full_version >= '3.8'}<=0.1.3 is available and the requested Python version (>=3.7)
      does not satisfy Python>=3.8, we can conclude that all versions of check-sdist{python_full_version >= '3.8'} are
      incompatible.
      And because you require check-sdist{python_full_version >= '3.8'}, we can conclude that your requirements are
      unsatisfiable.
alex commented 1 month ago

The bug described ^ appears to be fixed now.

BurntSushi commented 1 month ago

It looks like uv 0.4.5 fixed it:

[andrew@duff i6730]$ cat uv.toml
environments = ["python_version >= '3.10'", "python_version >= '3.8' and python_version < '3.10'", "python_version < '3.8'"]
[andrew@duff i6730]$ echo 'check-sdist; python_version >= "3.8"' | uv-0.4.4 pip compile --universal -p 3.7 -
warning: uv is only compatible with Python >=3.8, found Python 3.7.9
  x No solution found when resolving dependencies for split (python_full_version >= '3.8' and python_full_version < '3.10'):
  `-> Because only check-sdist{python_full_version >= '3.8'}<=0.1.3 is available and the requested Python version (>=3.7) does not satisfy Python>=3.8, we can conclude that all versions of check-sdist{python_full_version >= '3.8'}
      are incompatible.
      And because you require check-sdist{python_full_version >= '3.8'}, we can conclude that your requirements are unsatisfiable.
[andrew@duff i6730]$ echo 'check-sdist; python_version >= "3.8"' | uv-0.4.5 pip compile --universal -p 3.7 -
warning: uv is only compatible with Python >=3.8, found Python 3.7.9
Resolved 10 packages in 5ms
# This file was autogenerated by uv via the following command:
#    uv pip compile --universal -p 3.7 -
build==1.2.1 ; python_full_version >= '3.8'
    # via check-sdist
check-sdist==0.1.3 ; python_full_version >= '3.8'
colorama==0.4.6 ; python_full_version >= '3.8' and os_name == 'nt'
    # via build
importlib-metadata==8.4.0 ; python_full_version >= '3.8' and python_full_version < '3.10.2'
    # via build
importlib-resources==6.4.4 ; python_full_version == '3.8.*'
    # via check-sdist
packaging==24.1 ; python_full_version >= '3.8'
    # via build
pathspec==0.12.1 ; python_full_version >= '3.8'
    # via check-sdist
pyproject-hooks==1.1.0 ; python_full_version >= '3.8'
    # via build
tomli==2.0.1 ; python_full_version >= '3.8' and python_full_version < '3.11'
    # via
    #   build
    #   check-sdist
zipp==3.20.1 ; python_full_version >= '3.8' and python_full_version < '3.10.2'
    # via
    #   importlib-metadata
    #   importlib-resources

I'm actually not totally sure which commit fixed this. I haven't done a bisect. I see multiple possible candidates I think?

Also, the python_version < '0' marker is weird. It can never be true, so I wonder what leads to us writing that out. That might be worth investigating too. (I know we can just filter such things out towards the end, but I'd like to better understand why they show up in the first place.)

charliermarsh commented 1 month ago

I believe Konsti dropped those in his unreachable PRs, maybe?

BurntSushi commented 1 month ago

They are definitely still around (using the example in the OP):

[andrew@duff i6730]$ echo 'argcomplete; python_version >= "3.8"' | uv-main pip compile --universal -p 3.7 -
warning: uv is only compatible with Python >=3.8, found Python 3.7.9
Resolved 4 packages in 32ms
# This file was autogenerated by uv via the following command:
#    uv pip compile --universal -p 3.7 -
argcomplete==3.1.2 ; python_full_version >= '3.8'
importlib-metadata==6.7.0 ; python_full_version < '0'
    # via argcomplete
typing-extensions==4.7.1 ; python_full_version < '0'
    # via importlib-metadata
zipp==3.15.0 ; python_full_version < '0'
    # via importlib-metadata
charliermarsh commented 1 month ago

But I don’t see them in your reproduced snippet, and those changes came after this issue.

charliermarsh commented 1 month ago

Oh sorry, you re-ran there, I see. Do those packages appear if you use uv lock instead? We probably just need to apply whatever Konsti did in the lockfile to the universal output case \cc @konstin

konstin commented 1 month ago

I believe Konsti dropped those in his unreachable PRs, maybe?

The pruning currently only happens in the transformation from graph to uv.lock, we should also prune requirements.txt entries with a false marker.

charliermarsh commented 1 month ago

Gonna track that last part here: https://github.com/astral-sh/uv/issues/7196

charliermarsh commented 1 day ago

Should be fixed in the next release (see: https://github.com/astral-sh/uv/pull/8628).