Resolver error on upgrade

[chludwig-haufe]

When upgrading the (pinned) dependencies in requirements.in of a project with an existing requirements.txt, pip-compile fails with a resolver error. It claims that an indirect dependency (here: cachecontrol[filecache]<0.14.0,>=0.13.0) cannot be resolved even though a version (0.13.1) matching the constraint is listed among the considered versions.

pip-compile successfully creates a requirements.txt from the same requirements.in file (with the same pinned project dependencies), if I first delete the existing requirements.txt. However, I came across this error when renovateBot failed to update requirements.txt – that's a scenario where I cannot delete requirements.txt first.

Environment Versions

1. MacOS 13.5 (also reproduced under Ubuntu jammy)
2. Python version: 3.11.2 (also observed under Python 3.10 and 3.9)
3. pip version: 23.2.1
4. pip-tools version: 7.3.0

Steps to replicate

(pip-tools-venv) piptools  $ rm -f requirements.in requirements.txt
(pip-tools-venv) piptools  $ echo "poetry==1.5.1\n" > requirements.in
(pip-tools-venv) piptools  $ pip-compile --generate-hashes -q requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
(pip-tools-venv) piptools  $ grep '^cachecontrol' requirements.txt
cachecontrol[filecache]==0.12.14 \
(pip-tools-venv) piptools  $ echo "poetry==1.6.1\n" > requirements.in
(pip-tools-venv) piptools  $ pip-compile --generate-hashes -q requirements.in
  ERROR: Could not find a version that satisfies the requirement cachecontrol[filecache]<0.14.0,>=0.13.0 (from poetry) (from versions: 0.5, 0.6, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.10.0, 0.10.1, 0.10.2, 0.10.3, 0.10.4, 0.10.6, 0.10.7, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.11.4, 0.11.5, 0.11.6, 0.11.7, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.12.6, 0.12.7, 0.12.8, 0.12.9, 0.12.10, 0.12.11, 0.12.12, 0.12.13, 0.12.14, 0.13.0, 0.13.1rc0, 0.13.1)
Traceback (most recent call last):
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 439, in resolve
    raise ResolutionImpossible(self.state.backtrack_causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('cachecontrol[filecache]<0.14.0,>=0.13.0'), parent=LinkCandidate('https://files.pythonhosted.org/packages/7d/25/f3bfda3c458d114005af99441d009936b85a34a730aeb9cf57fb2630d9f7/poetry-1.6.1-py3-none-any.whl (from https://pypi.org/simple/poetry/) (requires-python:>=3.8,<4.0)'))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/piptools/scripts/compile.py", line 659, in cli
    results = resolver.resolve(max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
                  ^^^^^^^^^^^^^^^^^
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/Users/christoph.ludwig/tmp/piptools/pip-tools-venv/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 101, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: No matching distribution found for cachecontrol[filecache]<0.14.0,>=0.13.0
(pip-tools-venv) piptools  $ rm requirements.txt
(pip-tools-venv) piptools  $ pip-compile --generate-hashes -q requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.
(pip-tools-venv) piptools  $ grep '^cachecontrol' requirements.txt
cachecontrol[filecache]==0.13.1 \

Expected result

requirements.txt is updated, no need to delete requirements.txt first

Actual result

The resolver error shown if requirements.txt exists already generated for an older version of the project dependency.

[chludwig-haufe]

I ran pip-compile in the debugger and saw (I think) what's the problem:

• In piptools.resolver.BacktrackingResolver._do_resolve(), the first resolution attempt fails because compatible_existing_constraints contains an incompatible version of cachecontrol. So far that's expected.

• In the exception handler, however, cause_ireq_names is assigned the value {'cachecontrol[filecache]'}, including the extra specifier. However, the corresponding key in compatible_existing_constraints is 'cachecontrol' (without extra).

  Therefore, the handler cannot find and drop the incompatible dependency from compatible_existing_constraints (i.e., cause_existing_ireq is None) and re-raises the exception.

This happens with and without --strip-extras.

[webknjaz]

Does pip install --no-deps cachecontrol==0.13.1 -vvvvv succeed under the same OS+arch+Python? I think that the backtracking resolver rejects sdist-only releases that it can't build. Though, this one appears to have wheels so that's probably not it.

Have you tried without the hash generation? Also, maybe try with --upgrade.

[chludwig-haufe]

• pip install --no-deps cachecontrol==0.13.1 -vvvvv does work. (Tested inside an Ubuntu Jammy container with Python 3.11.)
• Dropping --generate-hashes from the command line does not make any difference.

• With --upgrade, the error goes away – unfortunately, this does not help me in the RenovateBot scenario.

  RenovateBot does use --upgrade if it creates a merge request for "lock file maintenance". (I.e., when it updates the indirect dependencies in requirements.txt.) It apparently does not use this option if it wants to update a pinned dependency in requirements.in. The only control that I have over the command line created by RenovateBot is that it keeps the option listed in requirements.txt.

Of course, I am not familiar with the codebase of pip-tools. But what I observed in the debugger looks like a bug to me:

• The existing constraints read from requirements.txt are initially kept. I assume the intention is to upgrade as few (indirect) dependencies as possible (unless --upgrade is specified).
• If there are incompatible constraints, than the BacktrackingResolver apparently should drop the offending package(s) from the existing constraints and try again. But the dict with the existing constraints uses "plain" package names (without extras) as keys, whereas the name of the incompatible package listed in the exception does include the extras. Therefore, the code does not find the relevant entry in the existing constraints.

I will try whether I can come up with a PR.