Closed sanderr closed 1 year ago
This is essentially by design, based on the comment you linked. I don't believe it will ever result in an incorrect resolution, although I guess that as you say it may cause some unnecessary backtracking. I think this is a fairly rare case, though - the situation you describe where the versions of the extra dependencies and the main package are tightly coupled seems unusual.
I don't think this is a bug, as it doesn't cause incorrect behaviour.
Having said that, if someone were to produce a PR to improve this, it would definitely be considered. But be aware, extras are extremely tricky to handle correctly, so this will not be simple to address.
the situation you describe where the versions of the extra dependencies and the main package are tightly coupled seems unusual.
I have to agree with that, I think this is a rare case. I do think similar issues can occur in more common scenarios but I expect the backtracking to be more contained there.
I assumed this to be an easy thing to address, I guess I was wrong about that. After reading the comment once more I think I understand why. I'll try to have a look at the current implementation and see if I can come up with something that doesn't break compatibility.
Am I correct in thinking that "inmanta-core==6.0.0" "inmanta-core[pytest-inmanta-extensions]"
is functionally equivalent to "inmanta-core[pytest-inmanta-extensions]==6.0.0”
.
Pip (or your own requirements build process) could have a step that "simplifies" user requirements before resolution even starts. This however would not improve things when such requirements are part of transitive dependencies, at that point applying such simplifications on requirements probably would be quite tricky.
I was considering the same and I think it could already be an improvement if a more generic solution can not be found. But the use case that led me to create this ticket is as you describe (the constraint on the package without extra is a transitive dependency), so indeed it is not feasible there.
I just realized this isn't actually affected by the unconventional coupling between the two packages I mentioned in the initial report. It occurs for any package with an extra if you want to install an older version:
in(tmp-37e9a3aa0746e4e) sander@bedevere:~/.virtualenvs/tmp-37e9a3aa0746e4e$ pip download resolvelib==0.3.0 resolvelib[examples]
Collecting resolvelib==0.3.0
Downloading resolvelib-0.3.0-py2.py3-none-any.whl (10 kB)
Collecting resolvelib[examples]
Downloading resolvelib-1.0.1-py2.py3-none-any.whl (17 kB)
Downloading resolvelib-1.0.0-py2.py3-none-any.whl (17 kB)
Downloading resolvelib-0.9.0-py2.py3-none-any.whl (16 kB)
Downloading resolvelib-0.8.1-py2.py3-none-any.whl (16 kB)
Downloading resolvelib-0.8.0-py2.py3-none-any.whl (15 kB)
Using cached resolvelib-0.7.1-py2.py3-none-any.whl (15 kB)
Downloading resolvelib-0.7.0-py2.py3-none-any.whl (15 kB)
Using cached resolvelib-0.6.0-py2.py3-none-any.whl (15 kB)
Downloading resolvelib-0.5.4-py2.py3-none-any.whl (12 kB)
Downloading resolvelib-0.5.3-py2.py3-none-any.whl (12 kB)
Downloading resolvelib-0.5.2-py2.py3-none-any.whl (12 kB)
Downloading resolvelib-0.5.1-py2.py3-none-any.whl (11 kB)
Downloading resolvelib-0.4.0-py2.py3-none-any.whl (11 kB)
Collecting requests
Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting html5lib
Using cached html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting packaging
Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting six>=1.9
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting webencodings
Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting charset-normalizer<4,>=2
Using cached charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Saved ./resolvelib-0.3.0-py2.py3-none-any.whl
Saved ./html5lib-1.1-py2.py3-none-any.whl
Saved ./packaging-23.0-py3-none-any.whl
Saved ./requests-2.28.2-py3-none-any.whl
Saved ./certifi-2022.12.7-py3-none-any.whl
Saved ./charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./idna-3.4-py3-none-any.whl
Saved ./six-1.16.0-py2.py3-none-any.whl
Saved ./urllib3-1.26.15-py2.py3-none-any.whl
Saved ./webencodings-0.5.1-py2.py3-none-any.whl
Successfully downloaded resolvelib html5lib packaging requests certifi charset-normalizer idna six urllib3 webencodings
This is of course a ridiculous example (it's just the first simple package I could think of that I know has extras) but I think it shows this is a more generic issue than discussed up till here. Any package with an extra that needs to maintain some sort of LTS on its older versions is susceptible to this.
Of course, this doesn't change the fact that the end result is still correct (unless a dev version is requested on one of the requirements, pip doesn't offer dev candidates for the other in that case) and that it would be difficult to handle correctly.
This can lead to huge backtracking without providing a clear error where the problem is. I was just hit by this in https://github.com/WeblateOrg/docker/pull/1788 (log at https://github.com/WeblateOrg/docker/actions/runs/4666399591/jobs/8260949502?pr=1788 or in details below).
The problem in this case is that backtracking went into many packages and ended up saying the issue is in celery constraints, while it is actually in Django ones (which the pull request is trying to upgrade).
Reproducer: pip install 'Django==4.2' 'Weblate==4.16.4' 'celery==5.2.7'
Fails after a lot of backtracking with:
ERROR: Cannot install celery==5.2.7, celery[redis]==5.2.3, celery[redis]==5.2.4 and celery[redis]==5.2.6 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested celery==5.2.7
celery[redis] 5.2.6 depends on celery 5.2.6 (from https://files.pythonhosted.org/packages/a0/ed/8a2e381aa9fa6fa5ac6891b0b472e927892f57a39842eff18cc917ceba57/celery-5.2.6-py3-none-any.whl (from https://pypi.org/simple/celery/) (requires-python:>=3.7,))
The user requested celery==5.2.7
celery[redis] 5.2.4 depends on celery 5.2.4 (from https://files.pythonhosted.org/packages/9d/a2/77d4bf33786dba473218067527ec40002cdaeed3599e5377e0f32143bba2/celery-5.2.4-py3-none-any.whl (from https://pypi.org/simple/celery/) (requires-python:>=3.7,))
The user requested celery==5.2.7
celery[redis] 5.2.3 depends on celery 5.2.3 (from https://files.pythonhosted.org/packages/1e/c2/52a01d3f53ddf57c80b011714dd63295c69426121d35d0ff41976b83506c/celery-5.2.3-py3-none-any.whl (from https://pypi.org/simple/celery/) (requires-python:>=3.7,))
While I'd actually expect something like this:
ERROR: Cannot install Django[argon2]<4.2 and >=3.2 and Django==4.2 because these package versions have conflicting dependencies.
The conflict is caused by:
Weblate 4.16.4 depends on Django[argon2]<4.2 and >=3.2
The user requested Django==4.2
This can lead to huge backtracking without providing a clear error where the problem is. I was just hit by this in WeblateOrg/docker#1788 (log at https://github.com/WeblateOrg/docker/actions/runs/4666399591/jobs/8260949502?pr=1788 or in details below).
The problem in this case is that backtracking went into many packages and ended up saying the issue is in celery constraints, while it is actually in Django ones (which the pull request is trying to upgrade).
Reproducer:
pip install 'Django==4.2' 'Weblate==4.16.4' 'celery==5.2.7'
Because this specific example hits a more general backtracking slowness rather than just OPs specific issue the amount of time backtracking should be significantly reduced in the next version of Pip.
And it in fact it is, I tested both the number of packages visited (I would normally check the difference between packages visited and packages installed as this is a better indicator of how much "wasted" time there is, but as it results in a resolution impossible I can't do that) and the wall clock time on a second attempt where Pip no longer needs to download any package because everything is cached:
Pip Version | Packages Visited | Wall Clock Time |
---|---|---|
Pip 23.0.1 | 136 | 179.78s |
Pip main (74133f8) | 121 | 25.78s |
Better news still in this example Pip main gives a much better ResolutionImpossible error, although I doubt this is guaranteed in general, this probably stems from resolvelib taking better1 steps when backtracking:
The conflict is caused by:
The user requested Django==4.2
crispy-bootstrap3 2022.1 depends on django>=3.2
django-appconf 1.0.3 depends on django
django-celery-beat 2.4.0 depends on Django<4.2 and >=2.2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
I have also encountered this behavior "in the wild," although it did not cause a major slowdown for us.
$ cat test.pip
pytest-cov==3.0.0
coverage==6.3.2
$ python --version
Python 3.8.13
$ python -m pip --version
pip 23.1.2 from XXX/virtualenv/lib/python3.8/site-packages/pip (python 3.8)
$ pip install -Ur test.pip
Requirement already satisfied: pytest-cov==3.0.0 in XXX/virtualenv/lib/python3.8/site-packages (from -r test.pip (line 1)) (3.0.0)
Requirement already satisfied: coverage==6.3.2 in XXX/virtualenv/lib/python3.8/site-packages (from -r test.pip (line 2)) (6.3.2)
Requirement already satisfied: pytest>=4.6 in XXX/virtualenv/lib/python3.8/site-packages (from pytest-cov==3.0.0->-r test.pip (line 1)) (7.1.2)
Collecting coverage[toml]>=5.2.1 (from pytest-cov==3.0.0->-r test.pip (line 1))
Using cached coverage-7.2.5-cp38-cp38-macosx_10_9_x86_64.whl (200 kB)
INFO: pip is looking at multiple versions of coverage[toml] to determine which version is compatible with other requirements. This could take a while.
Using cached coverage-7.2.4-cp38-cp38-macosx_10_9_x86_64.whl (200 kB)
Using cached coverage-7.2.3-cp38-cp38-macosx_10_9_x86_64.whl (199 kB)
Using cached coverage-7.2.2-cp38-cp38-macosx_10_9_x86_64.whl (199 kB)
Using cached coverage-7.2.1-cp38-cp38-macosx_10_9_x86_64.whl (199 kB)
Using cached coverage-7.2.0-cp38-cp38-macosx_10_9_x86_64.whl (199 kB)
Using cached coverage-7.1.0-cp38-cp38-macosx_10_9_x86_64.whl (198 kB)
Using cached coverage-7.0.5-cp38-cp38-macosx_10_9_x86_64.whl (197 kB)
INFO: pip is looking at multiple versions of coverage[toml] to determine which version is compatible with other requirements. This could take a while.
Using cached coverage-7.0.4-cp38-cp38-macosx_10_9_x86_64.whl (197 kB)
Using cached coverage-7.0.3-cp38-cp38-macosx_10_9_x86_64.whl (196 kB)
Using cached coverage-7.0.2-cp38-cp38-macosx_10_9_x86_64.whl (196 kB)
Using cached coverage-7.0.1-cp38-cp38-macosx_10_9_x86_64.whl (187 kB)
Using cached coverage-7.0.0-cp38-cp38-macosx_10_9_x86_64.whl (187 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
Using cached coverage-6.5.0-cp38-cp38-macosx_10_9_x86_64.whl (185 kB)
Using cached coverage-6.4.4-cp38-cp38-macosx_10_9_x86_64.whl (184 kB)
Using cached coverage-6.4.3-cp38-cp38-macosx_10_9_x86_64.whl (184 kB)
Using cached coverage-6.4.2-cp38-cp38-macosx_10_9_x86_64.whl (184 kB)
Using cached coverage-6.4.1-cp38-cp38-macosx_10_9_x86_64.whl (184 kB)
Using cached coverage-6.4-cp38-cp38-macosx_10_9_x86_64.whl (183 kB)
Using cached coverage-6.3.3-cp38-cp38-macosx_10_9_x86_64.whl (183 kB)
Requirement already satisfied: tomli in XXX/virtualenv/lib/python3.8/site-packages (from coverage==6.3.2->-r test.pip (line 2)) (2.0.1)
Requirement already satisfied: attrs>=19.2.0 in XXX/virtualenv/lib/python3.8/site-packages (from pytest>=4.6->pytest-cov==3.0.0->-r test.pip (line 1)) (21.4.0)
Requirement already satisfied: iniconfig in XXX/virtualenv/lib/python3.8/site-packages (from pytest>=4.6->pytest-cov==3.0.0->-r test.pip (line 1)) (2.0.0)
Requirement already satisfied: packaging in XXX/virtualenv/lib/python3.8/site-packages (from pytest>=4.6->pytest-cov==3.0.0->-r test.pip (line 1)) (21.3)
Requirement already satisfied: pluggy<2.0,>=0.12 in XXX/virtualenv/lib/python3.8/site-packages (from pytest>=4.6->pytest-cov==3.0.0->-r test.pip (line 1)) (1.0.0)
Requirement already satisfied: py>=1.8.2 in XXX/virtualenv/lib/python3.8/site-packages (from pytest>=4.6->pytest-cov==3.0.0->-r test.pip (line 1)) (1.11.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in XXX/virtualenv/lib/python3.8/site-packages (from packaging->pytest>=4.6->pytest-cov==3.0.0->-r test.pip (line 1)) (3.0.9)
FYI, I am currently working on this and I'm hopeful that I'll be able to open a pull request soon. My current implementation solves all current examples reported in this thread and all tests still succeed but it is naive in the sense that it hinders error reporting (e.g. in the case of pip install 'Django==4.2' 'Weblate==4.16.4' 'celery==5.2.7'
reported above). I do have some concrete ideas to improve on it.
Any idea when this will be released?
It has been released. Do you still have an issue with it?
@sanderr still having issues with it for gitlint via pre-commit but perhaps that's virtualenv.
Ah, there is actually a small bug in it that has been causing issues for gitlint, see #12372. Does that match with the issue you're having? I have a fix that seems to be working but I haven't had the time yet to add test cases and open a pull request. I'm hoping to get to that this weekend. I can't comment on when that would be released though.
@sanderr yes, that's exactly our issue too, thank you for your efforts.
Description
When a requirements list contains both
mypackage<some_constraint>
andmy_package[my_extra]
, the two seem to be treated separately, without applying the constraint tomy_package[my_extra]
. In the end,pip
does manage to find a correct set of packages but this may involve unnecessary backtracking that (I think) could be easily avoided.Consider the package
inmanta-core
. It has an extra namedpytest-inmanta-extensions
, which is strongly tied to it by version. It constraints thepytest-inmanta-extensions
package to be the same version asinmanta-core
itself (see here).I tried the following:
pip download "inmanta-core==6.0.0" "inmanta-core[pytest-inmanta-extensions]"
, expecting it to trivially find the appropriate versions. Alas, it seems to pick the trivial candidate forinmanta-core
, but then pick a different (incorrect) candidate forinmanta-core[pytest-inmanta-extesions]
. This later results in backtracking on the latter until it stumbles on the correct version. In this case it has to backtrack over some ten versions, trying it with dev versions (e.g. with--pre
and our internal index) is way worse.Since the package with and without extra have the same base, I would think constraints can be shared safely.
I think the cause may be related to https://github.com/pypa/pip/issues/11913#issuecomment-1491341333, but I haven't dug deep enough in the pip internals to know for sure.
Expected behavior
See description
pip version
23.0.1 and main (81f6a9f)
Python version
3.9.15
OS
Linux
How to Reproduce
pip download "inmanta-core==6.0.0" "inmanta-core[pytest-inmanta-extensions]"
Output
Code of Conduct