pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000 error

sreelathasankaranarayanan commented 1 month ago

Description

We are consistently see this error in our pip install jobs, can you please suggest if there is a way to reduce the backtracking of dependencies to prevent this error from happenning. We started noticing this problem consistently since Jun 6. Has there been any changes with pip 24.0 or setuptools 70.0.0 or a combination that needs to be considered?

We notice this problem in python version <3.11 and in python version 3.12 onwards we do not see this error.

ERROR: Exception: Traceback (most recent call last): File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper status = run_func(*args) File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper return func(self, options, args) File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run requirement_set = resolver.resolve( File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve result = self._result = resolver.resolve( File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve state = resolution.resolve(requirements, max_rounds=max_rounds) File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve raise ResolutionTooDeep(max_rounds) pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

Expected behavior

pip installation should succeed without resolution too deep errors

pip version

24.0

Python version

3.9

OS

Ubuntu 20.0

How to Reproduce

pip install --upgrade

Output

ERROR: Exception: Traceback (most recent call last): File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper status = run_func(*args) File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper return func(self, options, args) File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 377, in run requirement_set = resolver.resolve( File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve result = self._result = resolver.resolve( File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve state = resolution.resolve(requirements, max_rounds=max_rounds) File "/home/jenkins/.local/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve raise ResolutionTooDeep(max_rounds) pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

Code of Conduct

[X] I agree to follow the PSF Code of Conduct.

notatallshaw commented 1 month ago

Can you give a reproducible example? Very likely there's been a change in the dependency tree that's hitting a pathological use case. I might be able to provide an additional constraint on your requirements as a workaround to this error.

If you would like, I have a branch where I have tried to significantly reduce the chance of this happening, here is an example how how to install https://github.com/pypa/pip/issues/12305#issuecomment-1914080691. Feedback on if this branch helped or not would be helpful to me.

sreelathasankaranarayanan commented 1 month ago

Thanks @notatallshaw , let me give that a try, so you are suggesting to install pip from your branch. Is there any restriction on the version of setuptools to be used?

sreelathasankaranarayanan commented 1 month ago

@notatallshaw that worked like a charm! But how do we consume this, is it going to be merged to pip:main and available as a version of pip that we can directly install?

notatallshaw commented 1 month ago

My hope is eventually it will be merged into pip main, but I have not had chance to work on it recently, and getting it into pip requires multiple steps.

If you can give me your requirements that cause this issue for you, I can likely identify how to fix them so it works with regular pip.

sreelathasankaranarayanan commented 1 month ago

@notatallshaw our project has the following dependencies in setup.cfg for our project is internal to our org.

auditree-arboretum>=0.5.0 deprecated>=1.2.10 ibmcloud_tools[cos]>=2.36.0 python-dateutil<3.0.0 SoftLayer>=5.7.2 sdcclient>=0.10.0 pygithub kubernetes>=21.7.0 pytenable strenum ibm-platform-services ibm_cloud_sdk_core

Since yesterday we started to see our build failing with pip trying to resolve dependency by getting a faulty old version of pytenable (1.4.2), this halted our installation.

Then we tried to pin the version of pytenable to >=1.4.13, but this helped get rid of the failure with pytenable 1.4.2, but then we started to consistently see the pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000 errors and looking at the logs it is not always throwing this error after getting a certain package or while trying to check dependencies for a specific package.

We are using pip version 24.0 and setuptools 70.0.0, we tried to downgrade the versions of these tools as well, but to no avail. Really appreciate your help with this.

This is the dependency tree got by running pipdeptree from. my project deptree.txt

pfmoore commented 1 month ago

I just tried this, but there's no project called ibmcloud_tools on PyPI. Could you provide a full, reproducible example?

notatallshaw commented 1 month ago

I was able to reproduce by looking at the deptree and substituting in the non-public packages with their dependencies, I was testing on Python 3.11 Linux and I found I had to add a few extra lower bounds so pip didn't try to build a package too old that building just failed:

auditree-arboretum>=0.5.0
deprecated>=1.2.10
configparser
python-dateutil
PyYAML
requests
ilcli>=0.3.2
PyJWT>=2.6.0
python-dateutil
requests>=2.28.2
configparser
python-dateutil
PyYAML
requests
python-dateutil<3.0.0
SoftLayer>=5.7.2
sdcclient>=0.10.0
pygithub>=1.58.0
kubernetes>=21.7.0
pytenable>=1.4.12
strenum>=0.4.10 
ibm-platform-services>=0.30.3 
ibm_cloud_sdk_core>=3.16.1
GitPython
ibm-cos-sdk >=2.12.1
ibm-platform-services >=0.22.5

There's probably a simpler set of requirements to reproduce this, but it takes a long time to test reproducing.

First, let's check if this resolves quickly from two days ago, testing using pypi-timemachine I set the date to "2024-06-05", and indeed it resolved very quickly. So the only thing that changed was the ecosystem, not pip or setuptools.

Looking at what packages downloaded instead of using cached versions when I ran the version against "2024-06-05" the big thing that stood out to me was sdcclient which had a release on the 6th June.

If I put an upper bound on sdcclient sdcclient>=0.10.0,<0.17.2 then indeed it quickly resolves again! So this is our culprit but it is not satisfying to put an upper bound on it and leave it at that.

Comparing the metadata from the two releases:

There are two significant changes requests-toolbelt (>=0.9.1,<0.10.0) to requests-toolbelt (>=1.0.0,<2.0.0) and urllib3 (>=1.26.0,<2.0.0) to urllib3 (>=2.2.1,<3.0.0), we can find the likely culprit by swapping out sdcclient>=0.10.0 with either of these two. And indeed, the culprit reveals itself to be urllib3 (>=2.2.1,<3.0.0), swapping that in produces the error again.

A little further investigation revealed that ibm-cos-sdk-core always depends on some urllib3 <1.27 and therefore is not comatible with sdcclient 0.17.2.

So your solution for now is to:

Put an upper bound sdcclient<0.17.2
And, reach out to ibm-cos-sdk-core and ask them to remove urllib3 <1.27 unless they know for sure newer versions break their framework, and in general remind library authors not to eagerly put upper bounds on dependencies unless they have good reason (https://iscinumpy.dev/post/bound-version-constraints/) as it generally causes long term damage to the ecosystem
Politely ask sdcclient to loosen their requirements also, urllib3 1.x is still supported, are they actually already depending on 2.x features?

Hope that helps.

Pip is particularly bad at handeling these situations where it encounters a bad upper bound deep into the dependency tree. But this is a duplicate of https://github.com/pypa/pip/issues/12305 and hopefully we can eventually get pip much better.

sreelathasankaranarayanan commented 1 month ago

@notatallshaw Thanks a ton for helping debug the problem for us. It immensely helped us to understand the whole problem we faced wrt the sudden change in our project's behaviour in dependency resolution without any changes from our end. Agree that this is a duplicate of #12305 and I hope your fix for the same can make it to a future pip version soon.

pypa / pip