search

jazzband / pip-tools

A set of tools to keep your pinned Python dependencies fresh.
https://pip-tools.rtfd.io
BSD 3-Clause "New" or "Revised" License
7.69k stars 610 forks source link

ResolutionImpossible when constraints seem compatible #2121

Closed dabljues closed 3 weeks ago

dabljues commented 3 weeks ago

I have a project which uses pip-compile to generate requirements.txt and requirements-test.txt. I simply bump some package version in pyproject.toml and run these commands:

    pip-compile --generate-hashes --build-isolation --output-file=requirements.txt pyproject.toml
    pip-compile --generate-hashes --build-isolation --extra=dev --constraint=requirements.txt --output-file=requirements-test.txt pyproject.toml

The requirements-test.txt file is used for tests. All is run via tox.

When I try to bump responses package to a new version (0.23.1 -> 0.25.3), I get the following error:

pip-compile: commands[1]> pip-compile --generate-hashes --build-isolation --extra=dev --constraint=requirements.txt --output-file=requirements-test.txt pyproject.toml
  ERROR: Cannot install foo (pyproject.toml) because these package versions have conflicting dependencies.
Discarding requests==2.28.1 (from -r requirements-test.txt (line 297)) to proceed the resolution
  ERROR: Cannot install foo (pyproject.toml) because these package versions have conflicting dependencies.
Traceback (most recent call last):
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 316, in _backjump
    name, candidate = broken_state.mapping.popitem()
KeyError: 'dictionary is empty'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 434, in resolve
    success = self._backjump(causes)
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in _backjump
    raise ResolutionImpossible(causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('requests<3.0.0,>=2.24.0'), parent=LinkCandidate('<some_package_from_internal_artifactory> (from <internal_artifactory>) (requires-python:>=3.10)')), RequirementInformation(requirement=SpecifierRequirement('requests<3.0,>=2.30.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/12/24/93293d0be0db9da1ed8dfc5e6af700fdd40e8f10a928704dd179db9f03c1/responses-0.25.3-py3-none-any.whl (from https://pypi.org/simple/responses/) (requires-python:>=3.8)'))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dabljues/projects/foo/.tox/pip-compile/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/piptools/scripts/compile.py", line 470, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/home/dabljues/projects/foo/.tox/pip-compile/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 104, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

The responses library is in dev requirements of the package:

[project.optional-dependencies]
dev = [
    "responses == 0.25.3",
]

so it's obviously the 2nd command (with requirements-test.txt) that fails.

So, the "internal package" (redacted, sorry, let's call it bar) requires requests, and the responses package also requires them.

So, looking at the ResolutionImpossible message:

[RequirementInformation(requirement=SpecifierRequirement('requests<3.0.0,>=2.24.0'), parent=LinkCandidate('<some_package_from_internal_artifactory> (from <internal_artifactory>) (requires-python:>=3.10)')), RequirementInformation(requirement=SpecifierRequirement('requests<3.0,>=2.30.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/12/24/93293d0be0db9da1ed8dfc5e6af700fdd40e8f10a928704dd179db9f03c1/responses-0.25.3-py3-none-any.whl (from https://pypi.org/simple/responses/) (requires-python:>=3.8)'))]

well - it looks like those constraints are not conflicting. The constraints are >=, I'm using Python 3.10, serves both cases. These two packages in the exact versions mentioned here can be installed in a virtual environment, no problem.

requests are at version 2.28.1 in both requirements files. I know I can bump it and then it would work. However, in this case especially, requests is not even a direct dependency. So I would have to pin its version here in this project, just because some 3rd party library uses it. I also know I can use --upgrade, but it bumps all the dependencies, which is not desired - for example it can attempt to bump some library which dependencies could potentially collide with some other one's. I would expect that the first pip-compile would bump the requests in requirements.txt. If it was a direct dependency (like requests >= 2.30 directly in my project's pyproject.toml) - pip-compile would bump it. But since it's indirect - it doesn't?

Is it a bug? Or am I doing something wrong?

Environment Versions

  1. OS Type: Linux
  2. Python version: 3.10
  3. pip version: 23.3.1
  4. pip-tools version: 7.3.0
dabljues commented 3 weeks ago

I guess the best option I have is to remove requests==2.28.1 from requirements.txt and run pip-compile again.