Resolver preformance regression in 21.2

Description

As of 21.2, the resolver decides it needs to download every possible version, to determine which version is compatible with other requirements., for multiple dependencies.

Note that this works fine for some dependencies (coverage, pytest_cov), as it only downloads a few. For dependencies botocore and boto3, however, it downloads every possible version.

A minimal reproducible example can be found here: https://github.com/bblommers/pipresolverbug

Not sure what the offending dependency/installation method is - I could only reproduce it for this specific workflow.

The CI for this project runs this specific example for both 21.1 and 21.2. Version 21.1 completes in 15 seconds - 21.2 is still running after 20 minutes.

The CI logs can be found here: https://github.com/bblommers/pipresolverbug/actions/runs/1064350454

Expected behavior

To only download the latest version of a dependency

pip version

21.2

Python version

3.7

OS

Linux (Ubuntu)

How to Reproduce

git clone git@github.com:bblommers/pipresolverbug.git
pip install -r req.txt

Output

Collecting botocore>=1.12.201
  Using cached botocore-1.21.2-py3-none-any.whl (7.7 MB)
Collecting boto3>=1.9.201
  Using cached boto3-1.18.1-py3-none-any.whl (131 kB)
Collecting botocore>=1.12.201
  Using cached botocore-1.21.1-py3-none-any.whl (7.7 MB)
Collecting boto3>=1.9.201
  Using cached boto3-1.18.0-py3-none-any.whl (131 kB)
Collecting botocore>=1.12.201
  Using cached botocore-1.21.0-py3-none-any.whl (7.7 MB)
Collecting boto3>=1.9.201
  Using cached boto3-1.17.112-py2.py3-none-any.whl (131 kB)
Collecting s3transfer<0.5.0,>=0.4.0
  Using cached s3transfer-0.4.2-py2.py3-none-any.whl (79 kB)
Collecting botocore>=1.12.201
  Using cached botocore-1.20.112-py2.py3-none-any.whl (7.7 MB)
INFO: pip is looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while.
Collecting boto3>=1.9.201
  Using cached boto3-1.17.111-py2.py3-none-any.whl (131 kB)
Collecting botocore>=1.12.201

Code of Conduct

[X] I agree to follow the PSF Code of Conduct.

This is not a bug. Please kindly read this section of the documentation: https://pip.pypa.io/en/stable/user_guide/#dependency-resolution-backtracking

To propose improvements to the dependency resolution algorithm, please see #9187.

Reopening since I missed the part you said 21.1 worked reasonably well. I'll investigate whether we can bring back the 21.1 performance for you.

Collecting botocore>=1.12.201
  Using cached botocore-1.21.2-py3-none-any.whl (7.7 MB)
Collecting boto3>=1.9.201
  Using cached boto3-1.18.1-py3-none-any.whl (131 kB)
Collecting botocore>=1.12.201
  Using cached botocore-1.21.1-py3-none-any.whl (7.7 MB)
Collecting boto3>=1.9.201
  Using cached boto3-1.18.0-py3-none-any.whl (131 kB)
Collecting botocore>=1.12.201
  Using cached botocore-1.21.0-py3-none-any.whl (7.7 MB)

Looks like a state pollution issue!

Reopening since I missed the part you said 21.1 worked reasonably well. I'll investigate whether we can bring back the 21.1 performance for you.

Thanks @uranusjr. Checking the logs, it wasn't backtracking at all in 21.1 - it immediately accepted the latest dependency version (as it should). So to my untrained eyes, 21.2 is backtracking when it shouldn't.

Hopefully the repo and CI logs help in narrowing this down, but let me know if there's anything else I can do.

Looks like a state pollution issue!

I think it’s simply trying to find a boto3 version that specifies a botocore version that is compatible with another dependency. The situation is probablyt something like this

A package depends on one of the dependencies of botocore, and is incompatible with botocore 1.21.1 (but botocore 1.21.1 itself is not excluded).
Another package depends on boto3>=1.9.201. A lot of boto3 versions have a very specific botocore pin that's only satisfiable by botocore 1.21.1.
So the resolver tries each boto3 version and repeatedly
1. Try to find a compatible version of botocore (and find 1.21.1 compatible)
2. Try to pin botocore 1.21.1
3. Fail because botocore 1.21.1 contains an incompatibility with already pinned dependencies
4. Backtrack to unpin boto3

Step 3-ii. and 3-iii. can be optimised with tree pruning, but that still leaves 3-i. To have a useful optimisation, we need to find what exactly is causing botocore 1.21.1 to fail, and come up with something to avoid backtracking.

Checking the logs, it wasn't backtracking at all in 21.1 - it immediately accepted the latest dependency version (as it should).

If you look closely, 21.1 still backtracks, but on different packages (specifically coverage and pytest-cov). It's simply “more lucky” (lacking a better description) and only needed to backtrack a few versions before finding one that works, aided by the fact that both coverage and pytest-cov are significantly smaller in size than boto3 and botocore. So it's more complicated than you likely imagined 🙂

If you look closely, 21.1 still backtracks, but on different packages (specifically coverage and pytest-cov).

Yeah, I should have been more specific - I meant that 2.1 is not backtracking on botocore/boto3, so I'm surprised it needs to backtrack now with 21.2 - considering the botocore/boto3 versions are the same

So it's more complicated than you likely imagined slightly_smiling_face

It always is :slightly_smiling_face:

One of the difficulties here is that botocore has so many releases (1224 in my extract, but it's probably gone up since then). with the current limitations on the packaging infrastructure (getting metadata needs a download) that's always going to be an issue.

Less than 0.8% of packages on PyPI have more than 100 releases, so in principle it's entirely reasonable for the trade-offs that pip makes to favour projects with fewer releases. Unfortunately, projects like botocore are too popular to be considered "outliers" in a general sense.

I wonder if we could prioritise projects with fewer releases over those with more? Or is that something we've already tried?

On the other hand, given that it is only a few projects, maybe we should be looking at asking those projects to meet us half way? Maybe botocore/boto3 could yank older, unsupported versions? The support page isn't very clear on precisely which minor versions are supported for how long, but I can't imagine Amazon support all 1224 versions of botocore...

Didn't we try to resolve projects with fewer releases before projects with lots of releases? Or am I remembering it the other way round?

We only resolve packages with "tighter constraints" first, but don't actually count the number of releases. I think the main reason is there's currently no mechanism to do that; the information is of course available on the index page, but we don't have a way to bubble it up past PackageFinder.

Less than 0.8% of packages on PyPI have more than 100 releases, so in principle it's entirely reasonable for the trade-offs that pip makes to favour projects with fewer releases. Unfortunately, projects like botocore are too popular to be considered "outliers" in a general sense.

I wonder if we could prioritise projects with fewer releases over those with more? Or is that something we've already tried?

So while true most packages only have a few release, unfortunately looking at the top 10 packages on PyPi 50% have over 100 releases: botocore, awscli, boto3, requests, setuptools.

In the past (not sure if it's still true) setuptools has been identified in the resolver as a special case, I think it was to deprioritize checking for older versions? IMO I think it makes sense to add the other 4 packages listed here to that special case, but at the least add botocore, awcli, and boto3 as it looks like all 3 packages release daily.

What makes the pip resolver interesting about it's dependency resolution vs. other dependency resolver like conda is that exploring the dependency graph is a non-trivial cost. And it makes intuitive sense that packages with very frequent release (e.g. daily) will not change their dependency requirements every release and therefore trying to backtrack through those packages is going to be highly costly (this is my intuition and unsubstantiated by any data, I'll have a think if there's anyway I could empirically collect this information). But as @uranusjr points out this information currently isn't propagated to the pip resolver. So maybe it makes sense to special case very popular very commonly released packages?

My 2 cents.

Or, put more resources into PEP 658 or https://github.com/pypa/warehouse/issues/474, which should alleviate some of these pains IIUC?

It's also interesting to note that we never have these issues with setuptools or requests. I don't know whether there's something about the release policies of the other three (which AIUI are all Amazon interface APIs, is that right?) that means there's more of a tendency for dependencies to conflict? Do they release breaking changes more often, so consumers have to pin versions relatively tightly, causing conflicts? Do they advise tight pins in their documentation, causing consumers to do things that interact badly with pip's resolution algorithms?

Note - I'm not trying to "punish" these packages. There's a real problem here and it needs fixing, but step one of that should be to understand why the problem is occurring, and I (and likely the other pip devs as well) don't understand why there's such an issue with those projects in particular. My first thought was "lots of versions". Thanks for the information that it's not that simple, I hadn't realised requests and setuptools also had hundreds of releases. So let's try to dig deeper.

I can't actually tell in many of these reports whether there is a conflict that means the install will fail in the end, or whether everything is actually fine, and pip just takes ages to find the correct solution. (Not surprising, we can't necessarily expect people to spend hours waiting for a run to complete, possibly multiple times, just so they can raise a bug report). My impression is that it's typically the former, and "getting an error more quickly" is generally the objective here.

One problem is that the pip developers also have limited time. I've personally been interested in trying to work out what's going on here for months, but have never had the spare time to download all of those versions for their metadata, let alone to do a proper analysis of the dependency graph. So we're very reliant on the people with the issue helping us with whatever insights they can supply.

put more resources into PEP 658 or pypa/warehouse#474

Yep, if we had more resources available to put into those 🙁

Even if we did, analyzing all of the versions would still be slow (orders of magnitude faster if we could just download the metadata, so probably not an issue, but nevertheless still slower than the "average" package). So trying to address the underlying problem of handling huge dependency trees in a more user-friendly manner is still worth doing.

It's also interesting to note that we never have these issues with setuptools or requests.

FYI there were lots of issues with setuptools when the new pip resolver was released. As a mentioned it was special cased, found the relevant PR: https://github.com/pypa/pip/pull/9249

It looks to me like an argument could be made for adding botocore, awscli, and boto3 to that same special case as a quick dirty fix until more resources are available from either Amazon or pip to come up with a clean generalized solution.

I can't actually tell in many of these reports whether there is a conflict that means the install will fail in the end, or whether everything is actually fine, and pip just takes ages to find the correct solution. (Not surprising, we can't necessarily expect people to spend hours waiting for a run to complete, possibly multiple times, just so they can raise a bug report). My impression is that it's typically the former, and "getting an error more quickly" is generally the objective here.

OP says in this issue that 21.1 came to a resolution relatively quickly and 21.2 does not. Hence in this case I think this is an issue of the latter not the former.

As a mentioned it was special cased, found the relevant PR: #9249

Thanks, I'd forgotten that. Does adding botocore/boto3 to that list help with this issue?

OP says in this issue that 21.1 came to a resolution relatively quickly and 21.2 does not.

Doh. I missed that inference. So yes looks like this is taking a long time to find a valid solution.

Thanks, I'd forgotten that. Does adding botocore/boto3 to that list help with this issue?

If no one else does by this evening (EST) I'll attempt to reproduce the issue and then try adding those packages to the list and report the results.

Just another report here - we are running into problems with the resolver taking a VERY LONG time, but if we revert to pip version 21.1.3 the issue goes away.

It's also interesting to note that we never have these issues with setuptools or requests. I don't know whether there's something about the release policies of the other three (which AIUI are all Amazon interface APIs, is that right?) that means there's more of a tendency for dependencies to conflict?

FYI there were lots of issues with setuptools when the new pip resolver was released. As a mentioned it was special cased, found the relevant PR: #9249

It’s definitely related to their release poliocies. When the setuptools hack was added, the resolver’s logic was considerable simpler than it is now. setuptools almost never needs to be downloaded more than once (it has awesome backward compatibility and practically nobody pins it with a < or ==, so whatever version downloaded first try is usually good enough), and with the resolver logic now, I am pretty sure the performance degradation would be barely noticable if we remove the hack now. But the logic is already there, so we take advantage of what we can get.

Those AWS packages are different. They are quite tightly coupled together and have many tight pins (either == or >X, <Y ones with only a few versions between), so conflicts happen far more often. They usually tend to happen deeper in the graph (people don’t generally depend on say botocore directly, but trasitively through boto3 or awscli or something else), which makes backtracking even more likely. It also does not help that botocore is about 9x in size to setuptools.

I think the idea to delay resolving projects with more releases is likely worth exploring. Having more releases is not an issue by itself, only when those releases have more stricter dependencies, but delaying packages with loose dependencies is usually inconsequential. The packages negatively affected would be those with not too many releases (say around 20) but with tight dependencies; backtracking through those 20 files is still quite painful. However, we can only know if a package version has tight dependencies after we actually download it, so there’s probably not much we can do for them.

@ReneeErnst Please post what you are trying to install and the output you get. We can’t know what needs to change without knowing the concrete situation.

@uranusjr Sorry about that. We are installing the packages you are discussing above (boto3, awscli). Release 21.1 was working fine, but we experienced the same issue as bblomers with 21.2. Sounds like the example they provided covers our issue as well.

I followed OPs steps and tested botocore and boto3 triggering the "delay_this" variable:

Pip 21.1.3: 8 seconds
Pip 21.2.1: 1084 seconds
Pip 21.2.1 with botocore delayed: 82 seconds
Pip 21.2.1 with boto3 delayed: 57 seconds
Pip 21.2.1 with boto3 and botocore delayed: 8 seconds

All timings are using cached packages only (in the worst case here ~3 GBs of packages are downloaded) and are rounded to the nearest second, running on Windows 10, Python 3.9.6, AMD Ryzen 5900X, and a PCIe Gen 4 SSD.

This was the line of code I changed: https://github.com/notatallshaw/pip/blob/8666868b406b6250ae60e9716c95d74385fd3db4/src/pip/_internal/resolution/resolvelib/provider.py#L133

Our github actions setup step which used pip was failing after running for over 4 hours. capped pip at 21.1.3 and in is now back to 15 seconds.

@Christian-B Please post what you are trying to install and the output you get. We can’t know what needs to change without knowing the concrete situation.

Searching for quantities>=0.12.1 Reading https://pypi.org/simple/quantities/ Downloading https://files.pythonhosted.org/packages/2b/4f/2e8ce7d6c16fb07c43036f8539962322b2bf241e7397e87f318a1aa2f7c4/quantities-0.12.4.tar.gz#sha256=a33d636d1870c9e1127631185d89b0105a49f827d6aacd44ad9d8f151f331d8b Best match: quantities 0.12.4 Processing quantities-0.12.4.tar.gz Writing /tmp/easy_install-es421mrw/quantities-0.12.4/setup.cfg Running quantities-0.12.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-es421mrw/quantities-0.12.4/egg-dist-tmp-u3cnastb zip_safe flag not set; analyzing archive contents... error: Setup script exited with interrupted

Searching for quantities>=0.12.1 Reading https://pypi.org/simple/quantities/ Downloading https://files.pythonhosted.org/packages/2b/4f/2e8ce7d6c16fb07c43036f8539962322b2bf241e7397e87f318a1aa2f7c4/quantities-0.12.4.tar.gz#sha256=a33d636d1870c9e1127631185d89b0105a49f827d6aacd44ad9d8f151f331d8b Best match: quantities 0.12.4 Processing quantities-0.12.4.tar.gz Writing /tmp/easy_install-es421mrw/quantities-0.12.4/setup.cfg Running quantities-0.12.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-es421mrw/quantities-0.12.4/egg-dist-tmp-u3cnastb zip_safe flag not set; analyzing archive contents... error: Setup script exited with interrupted

FYI I just tried reproducing with Python 3.9.6 and Pip 21.2.1 in a clean environment running the command python -m pip install quantities>=0.12.1 and it installed in less than 5 seconds. I don't think this is the same issue and you will need to give more details (preferably in a new issue) to make a reproducible test case.

This new resolver just destroyed our CI process, which went from 4 minutes to timing out at 6 hours.

This new resolver just destroyed our CI process, which went from 4 minutes to timing out at 6 hours.

Here too... unfortunately "ate" lots of GitHub actions hours for us 👎🏼

Or, put more resources into PEP 658 or pypa/warehouse#474, which should alleviate some of these pains IIUC?

So just quickly coming back to this and looking at the benchmark results:

I followed OPs steps and tested botocore and boto3 triggering the "delay_this" variable:

* Pip 21.1.3: 8 seconds

* Pip 21.2.1: 1084 seconds

* Pip 21.2.1 with botocore delayed: 82 seconds

* Pip 21.2.1 with boto3 delayed: 57 seconds

* Pip 21.2.1 with boto3 and botocore delayed: 8 seconds

I don't think PEP 658 or pypa/warehouse#474 would fully resolve this specific performance regression as this benchmark was done with all packages cached and the majority of the 1084 seconds was spent resolving the dependency graph.

Though obviously the additional impact of downloading ~3 GBs of packages is a significant performance issue for many, still it's only part of the issue here.

I want to emphasis again, please post what you are installing and the output you get. Dropping a comment without any details won’t get your issue fixed. We are developers here, not psychics.

I don't think PEP 658 or pypa/warehouse#474 would fully resolve this specific performance regression

Agreed. Regardless of the whole issue of downloading, simply resolving a combination of thousands of dependency rules is hard, and cannot be quick in all cases. Dependency resolution is an NP-hard problem. Whatever we do will always be fast for some cases and slow for others, there's simply no way to avoid that if we want to get correct results.

What we'll improve by having static metadata available is the constant factor, so we'll be able to ignore the fundamentally exponential complexity for a while longer. How much longer probably depends on whether botocore/boto3 continue publishing versions at breakneck speed. Give it a week or two, maybe 😉

I want to emphasis again, please post what you are installing and the output you get. Dropping a comment without any details won’t get your issue fixed. We are developers here, not psychics.

The problem happens with any and all packages that aren't pinned to a specific version. Some examples of long-running ones are billiard, cffi, certifi, and autobahn. If you use > or < it will download every single version of those packages.

In other threads the response has been "pin specific versions" and if that's the solution to a problem that didn't exist previously, then just remove any operators except ==

The problem happens with any and all packages that aren't pinned to a specific version.

I'm not part of pip team but I'd like to point out this isn't true, I have lots of requirements that don't pin a specific version of a package and I don't have this issue.

As already stated by others if you don't post specifically what you're doing (giving the command and requirements) then your problem probably won't be helped.

This small snippet of the resolution process has taken 9 hours so far:

Collecting coverage[toml]>=5.2.1
  Downloading coverage-5.4-cp38-cp38-manylinux2010_x86_64.whl (245 kB)
  Downloading coverage-5.3.1-cp38-cp38-manylinux2010_x86_64.whl (245 kB)
  Downloading coverage-5.3-cp38-cp38-manylinux1_x86_64.whl (230 kB)
  Downloading coverage-5.2.1-cp38-cp38-manylinux1_x86_64.whl (231 kB)
INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while.
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
  Downloading chardet-3.0.3-py2.py3-none-any.whl (133 kB)
  Downloading chardet-3.0.2-py2.py3-none-any.whl (133 kB)
INFO: pip is looking at multiple versions of babel to determine which version is compatible with other requirements. This could take a while.
Collecting babel!=2.0,>=1.3
  Downloading Babel-2.9.0-py2.py3-none-any.whl (8.8 MB)
INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while.
  Downloading Babel-2.8.1-py2.py3-none-any.whl (8.6 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading Babel-2.8.0-py2.py3-none-any.whl (8.6 MB)
  Downloading Babel-2.7.0-py2.py3-none-any.whl (8.4 MB)
  Downloading Babel-2.6.0-py2.py3-none-any.whl (8.1 MB)
  Downloading Babel-2.5.3-py2.py3-none-any.whl (6.8 MB)
  Downloading Babel-2.5.2-py2.py3-none-any.whl (6.8 MB)
INFO: pip is looking at multiple versions of babel to determine which version is compatible with other requirements. This could take a while.
  Downloading Babel-2.5.1-py2.py3-none-any.whl (6.8 MB)
  Downloading Babel-2.5.0-py2.py3-none-any.whl (6.8 MB)
  Downloading Babel-2.4.0-py2.py3-none-any.whl (6.8 MB)
  Downloading Babel-2.3.4-py2.py3-none-any.whl (7.1 MB)
  Downloading Babel-2.3.3-py2.py3-none-any.whl (7.1 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading Babel-2.3.2-py2.py3-none-any.whl (7.1 MB)
  Downloading Babel-2.3.1-py2.py3-none-any.whl (7.2 MB)
  Downloading Babel-2.3.0-py2.py3-none-any.whl (97 kB)
  Downloading Babel-2.2.0-py2.py3-none-any.whl (6.5 MB)
  Downloading Babel-2.1.1-py2.py3-none-any.whl (3.6 MB)
  Downloading Babel-1.3.tar.gz (3.4 MB)
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
Collecting attrs>=17.4.0
  Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
  Downloading attrs-20.2.0-py2.py3-none-any.whl (48 kB)
  Downloading attrs-20.1.0-py2.py3-none-any.whl (49 kB)
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
  Downloading attrs-19.2.0-py2.py3-none-any.whl (40 kB)
  Downloading attrs-19.1.0-py2.py3-none-any.whl (35 kB)
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
  Downloading attrs-18.2.0-py2.py3-none-any.whl (34 kB)
  Downloading attrs-18.1.0-py2.py3-none-any.whl (28 kB)
  Downloading attrs-17.4.0-py2.py3-none-any.whl (31 kB)
INFO: pip is looking at multiple versions of astroid to determine which version is compatible with other requirements. This could take a while.
Collecting astroid<2.0,>=1.6
  Downloading astroid-1.6.5-py2.py3-none-any.whl (293 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading astroid-1.6.4-py2.py3-none-any.whl (290 kB)
  Downloading astroid-1.6.3-py2.py3-none-any.whl (289 kB)
  Downloading astroid-1.6.2-py2.py3-none-any.whl (291 kB)
  Downloading astroid-1.6.1-py2.py3-none-any.whl (288 kB)
  Downloading astroid-1.6.0-py2.py3-none-any.whl (288 kB)
INFO: pip is looking at multiple versions of asgiref to determine which version is compatible with other requirements. This could take a while.
Collecting asgiref<4,>=3.2.10
  Downloading asgiref-3.4.1-py3-none-any.whl (25 kB)
INFO: pip is looking at multiple versions of astroid to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading asgiref-3.4.0-py3-none-any.whl (25 kB)
  Downloading asgiref-3.3.4-py3-none-any.whl (22 kB)
  Downloading asgiref-3.3.3-py3-none-any.whl (22 kB)
  Downloading asgiref-3.3.2-py3-none-any.whl (22 kB)
INFO: pip is looking at multiple versions of amqp to determine which version is compatible with other requirements. This could take a while.
Collecting amqp<3.0,>=2.4.0
  Downloading amqp-2.6.1-py2.py3-none-any.whl (48 kB)
INFO: pip is looking at multiple versions of asgiref to determine which version is compatible with other requirements. This could take a while.
  Downloading amqp-2.6.0-py2.py3-none-any.whl (47 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading amqp-2.5.2-py2.py3-none-any.whl (49 kB)
  Downloading amqp-2.5.1-py2.py3-none-any.whl (49 kB)

The requirements.txt to cause it to happen:

  wheel==0.30.0
django>=3.2,<3.3 
six==1.15.0
django-environ==0.4.4
django-crispy-forms==1.7.0
django-model-utils==3.0.0
Pillow==8.2.0
argon2-cffi==20.1.0
django-allauth==0.34.0
psycopg2==2.8.5
awesome-slugify==1.6.5
pytz==2017.3
django-redis==4.12.1
redis==3.5.3
django-cors-headers==3.5.0
djangorestframework==3.12.2
djangorestframework-jwt==1.11.0
requests==2.25.1
newrelic
cryptography==3.4.6
kombu==4.3.0
celery==4.2.1
django-celery-results==1.0.4
social-auth-app-django==3.1.0
python-jose==3.0.1
elasticsearch_dsl==7.0.0
django-elasticsearch-dsl==7.1.4
django-elasticsearch-dsl-drf==0.20.9
channels==3.0.3
channels_redis==3.2.0
daphne==3.0.1
sorl-thumbnail==12.7.0
sorl-thumbnail-serializer-field==0.2.1
auth0-python==3.6.1
coreapi==2.3.3
hvac==0.10.1
django-debug-toolbar==2.2.1
django-extensions==2.2.9
confluent-kafka>=1.7,<1.8

The requirements.txt to cause it to happen:

I tried reproducing your issue on Linux with Python 3.9.6 and Pip 21.2.1, in my testing it took less than 11 seconds to install the requirements you provided:

$ python -V
Python 3.9.6

$ python -m pip -V
pip 21.2.1 from .../lib/python3.9/site-packages/pip (python 3.9)

$ cat req.txt
wheel==0.30.0
django>=3.2,<3.3
six==1.15.0
django-environ==0.4.4
django-crispy-forms==1.7.0
django-model-utils==3.0.0
Pillow==8.2.0
argon2-cffi==20.1.0
django-allauth==0.34.0
psycopg2==2.8.5
awesome-slugify==1.6.5
pytz==2017.3
django-redis==4.12.1
redis==3.5.3
django-cors-headers==3.5.0
djangorestframework==3.12.2
djangorestframework-jwt==1.11.0
requests==2.25.1
newrelic
cryptography==3.4.6
kombu==4.3.0
celery==4.2.1
django-celery-results==1.0.4
social-auth-app-django==3.1.0
python-jose==3.0.1
elasticsearch_dsl==7.0.0
django-elasticsearch-dsl==7.1.4
django-elasticsearch-dsl-drf==0.20.9
channels==3.0.3
channels_redis==3.2.0
daphne==3.0.1
sorl-thumbnail==12.7.0
sorl-thumbnail-serializer-field==0.2.1
auth0-python==3.6.1
coreapi==2.3.3
hvac==0.10.1
django-debug-toolbar==2.2.1
django-extensions==2.2.9
confluent-kafka>=1.7,<1.8

$ time python -m pip install -r req.txt
Collecting wheel==0.30.0
  Using cached wheel-0.30.0-py2.py3-none-any.whl (49 kB)
Collecting django<3.3,>=3.2
  Using cached Django-3.2.5-py3-none-any.whl (7.9 MB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting django-environ==0.4.4
  Using cached django_environ-0.4.4-py2.py3-none-any.whl (24 kB)
Collecting django-crispy-forms==1.7.0
  Using cached django_crispy_forms-1.7.0-py2.py3-none-any.whl (104 kB)
Collecting django-model-utils==3.0.0
  Using cached django_model_utils-3.0.0-py2.py3-none-any.whl
Collecting Pillow==8.2.0
  Using cached Pillow-8.2.0-cp39-cp39-manylinux1_x86_64.whl (3.0 MB)
Collecting argon2-cffi==20.1.0
  Using cached argon2_cffi-20.1.0-cp35-abi3-manylinux1_x86_64.whl (97 kB)
Collecting django-allauth==0.34.0
  Using cached django_allauth-0.34.0-py3-none-any.whl
Collecting psycopg2==2.8.5
  Using cached psycopg2-2.8.5-cp39-cp39-linux_x86_64.whl
Collecting awesome-slugify==1.6.5
  Using cached awesome_slugify-1.6.5-py3-none-any.whl
Collecting pytz==2017.3
  Using cached pytz-2017.3-py2.py3-none-any.whl (511 kB)
Collecting django-redis==4.12.1
  Using cached django_redis-4.12.1-py3-none-any.whl (19 kB)
Collecting redis==3.5.3
  Using cached redis-3.5.3-py2.py3-none-any.whl (72 kB)
Collecting django-cors-headers==3.5.0
  Using cached django_cors_headers-3.5.0-py3-none-any.whl (11 kB)
Collecting djangorestframework==3.12.2
  Using cached djangorestframework-3.12.2-py3-none-any.whl (957 kB)
Collecting djangorestframework-jwt==1.11.0
  Using cached djangorestframework_jwt-1.11.0-py2.py3-none-any.whl (13 kB)
Collecting requests==2.25.1
  Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting newrelic
  Using cached newrelic-6.4.4.161-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (697 kB)
Collecting cryptography==3.4.6
  Using cached cryptography-3.4.6-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
Collecting kombu==4.3.0
  Using cached kombu-4.3.0-py2.py3-none-any.whl (183 kB)
Collecting celery==4.2.1
  Using cached celery-4.2.1-py2.py3-none-any.whl (401 kB)
Collecting django-celery-results==1.0.4
  Using cached django_celery_results-1.0.4-py2.py3-none-any.whl (16 kB)
Collecting social-auth-app-django==3.1.0
  Using cached social_auth_app_django-3.1.0-py3-none-any.whl (23 kB)
Collecting python-jose==3.0.1
  Using cached python_jose-3.0.1-py2.py3-none-any.whl (25 kB)
Collecting elasticsearch_dsl==7.0.0
  Using cached elasticsearch_dsl-7.0.0-py2.py3-none-any.whl (48 kB)
Collecting django-elasticsearch-dsl==7.1.4
  Using cached django_elasticsearch_dsl-7.1.4-py2.py3-none-any.whl (19 kB)
Collecting django-elasticsearch-dsl-drf==0.20.9
  Using cached django_elasticsearch_dsl_drf-0.20.9-py2.py3-none-any.whl (146 kB)
Collecting channels==3.0.3
  Using cached channels-3.0.3-py3-none-any.whl (38 kB)
Collecting channels_redis==3.2.0
  Using cached channels_redis-3.2.0-py2.py3-none-any.whl (14 kB)
Collecting daphne==3.0.1
  Using cached daphne-3.0.1-py3-none-any.whl (26 kB)
Collecting sorl-thumbnail==12.7.0
  Using cached sorl_thumbnail-12.7.0-py3-none-any.whl (41 kB)
Collecting sorl-thumbnail-serializer-field==0.2.1
  Using cached sorl_thumbnail_serializer_field-0.2.1-py2.py3-none-any.whl (14 kB)
Collecting auth0-python==3.6.1
  Using cached auth0_python-3.6.1-py2.py3-none-any.whl (66 kB)
Collecting coreapi==2.3.3
  Using cached coreapi-2.3.3-py2.py3-none-any.whl (25 kB)
Collecting hvac==0.10.1
  Using cached hvac-0.10.1-py2.py3-none-any.whl (119 kB)
Collecting django-debug-toolbar==2.2.1
  Using cached django_debug_toolbar-2.2.1-py3-none-any.whl (200 kB)
Collecting django-extensions==2.2.9
  Using cached django_extensions-2.2.9-py2.py3-none-any.whl (217 kB)
Collecting confluent-kafka<1.8,>=1.7
  Using cached confluent_kafka-1.7.0-cp39-cp39-manylinux2010_x86_64.whl (2.7 MB)
Collecting cffi>=1.0.0
  Using cached cffi-1.14.6-cp39-cp39-manylinux1_x86_64.whl (405 kB)
Collecting python3-openid>=3.0.8
  Using cached python3_openid-3.2.0-py3-none-any.whl (133 kB)
Collecting requests-oauthlib>=0.3.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting Unidecode<0.05,>=0.04.14
  Using cached Unidecode-0.04.21-py2.py3-none-any.whl (228 kB)
Collecting regex
  Using cached regex-2021.7.6-cp39-cp39-manylinux2014_x86_64.whl (733 kB)
Collecting PyJWT<2.0.0,>=1.5.2
  Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: certifi>=2017.4.17 in ./miniforge3/envs/pip212/lib/python3.9/site-packages (from requests==2.25.1->-r req.txt (line 18)) (2021.5.30)
Collecting chardet<5,>=3.0.2
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting amqp<3.0,>=2.4.0
  Using cached amqp-2.6.1-py2.py3-none-any.whl (48 kB)
Collecting billiard<3.6.0,>=3.5.0.2
  Using cached billiard-3.5.0.5-py3-none-any.whl
Collecting social-auth-core>=1.2.0
  Using cached social_auth_core-4.1.0-py3-none-any.whl (333 kB)
Collecting future<1.0
  Using cached future-0.18.2-py3-none-any.whl
Collecting rsa
  Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting ecdsa<1.0
  Using cached ecdsa-0.17.0-py2.py3-none-any.whl (119 kB)
Collecting elasticsearch<8.0.0,>=7.0.0
  Using cached elasticsearch-7.13.4-py2.py3-none-any.whl (356 kB)
Collecting python-dateutil
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting django-nine>=0.2
  Using cached django_nine-0.2.4-py2.py3-none-any.whl (25 kB)
Collecting asgiref<4,>=3.2.10
  Using cached asgiref-3.4.1-py3-none-any.whl (25 kB)
Collecting aioredis~=1.0
  Using cached aioredis-1.3.1-py3-none-any.whl (65 kB)
Collecting msgpack~=1.0
  Using cached msgpack-1.0.2-cp39-cp39-manylinux1_x86_64.whl (294 kB)
Collecting autobahn>=0.18
  Using cached autobahn-21.3.1-py2.py3-none-any.whl (495 kB)
Collecting twisted[tls]>=18.7
  Using cached Twisted-21.7.0-py3-none-any.whl (3.1 MB)
Collecting coreschema
  Using cached coreschema-0.0.4-py3-none-any.whl
Collecting itypes
  Using cached itypes-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting uritemplate
  Using cached uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting sqlparse>=0.2.0
  Using cached sqlparse-0.4.1-py3-none-any.whl (42 kB)
Collecting async-timeout
  Using cached async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting hiredis
  Using cached hiredis-2.0.0-cp39-cp39-manylinux2010_x86_64.whl (85 kB)
Collecting vine<5.0.0a1,>=1.1.3
  Using cached vine-1.3.0-py2.py3-none-any.whl (14 kB)
Collecting hyperlink>=21.0.0
  Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
Collecting txaio>=21.2.1
  Using cached txaio-21.2.1-py2.py3-none-any.whl (30 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting defusedxml
  Using cached defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
Collecting social-auth-core>=1.2.0
  Using cached social_auth_core-4.0.3-py3-none-any.whl (328 kB)
  Using cached social_auth_core-4.0.2-py3-none-any.whl (330 kB)
Collecting zope.interface>=4.4.2
  Using cached zope.interface-5.4.0-cp39-cp39-manylinux2010_x86_64.whl (255 kB)
Collecting constantly>=15.1
  Using cached constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting typing-extensions>=3.6.5
  Using cached typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Collecting attrs>=19.2.0
  Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting Automat>=0.8.0
  Using cached Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting incremental>=21.3.0
  Using cached incremental-21.3.0-py2.py3-none-any.whl (15 kB)
Collecting service-identity>=18.1.0
  Using cached service_identity-21.1.0-py2.py3-none-any.whl (12 kB)
Collecting pyopenssl>=16.0.0
  Using cached pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Collecting pyasn1-modules
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Requirement already satisfied: setuptools in ./miniforge3/envs/pip212/lib/python3.9/site-packages (from zope.interface>=4.4.2->twisted[tls]>=18.7->daphne==3.0.1->-r req.txt (line 31)) (49.6.0.post20210108)
Collecting jinja2
  Using cached Jinja2-3.0.1-py3-none-any.whl (133 kB)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.0.1-cp39-cp39-manylinux2010_x86_64.whl (30 kB)
Installing collected packages: pycparser, six, pyasn1, idna, cffi, attrs, zope.interface, typing-extensions, pyasn1-modules, incremental, hyperlink, cryptography, constantly, Automat, vine, urllib3, txaio, twisted, service-identity, pyopenssl, chardet, sqlparse, requests, pytz, python-dateutil, oauthlib, MarkupSafe, elasticsearch, defusedxml, autobahn, asgiref, amqp, requests-oauthlib, python3-openid, PyJWT, kombu, jinja2, hiredis, elasticsearch-dsl, django, daphne, billiard, async-timeout, uritemplate, Unidecode, sorl-thumbnail, social-auth-core, rsa, regex, redis, msgpack, itypes, future, ecdsa, djangorestframework, django-nine, django-elasticsearch-dsl, coreschema, channels, celery, aioredis, wheel, sorl-thumbnail-serializer-field, social-auth-app-django, python-jose, psycopg2, Pillow, newrelic, hvac, djangorestframework-jwt, django-redis, django-model-utils, django-extensions, django-environ, django-elasticsearch-dsl-drf, django-debug-toolbar, django-crispy-forms, django-cors-headers, django-celery-results, django-allauth, coreapi, confluent-kafka, channels-redis, awesome-slugify, auth0-python, argon2-cffi
  Attempting uninstall: wheel
    Found existing installation: wheel 0.36.2
    Uninstalling wheel-0.36.2:
      Successfully uninstalled wheel-0.36.2
Successfully installed Automat-20.2.0 MarkupSafe-2.0.1 Pillow-8.2.0 PyJWT-1.7.1 Unidecode-0.4.21 aioredis-1.3.1 amqp-2.6.1 argon2-cffi-20.1.0 asgiref-3.4.1 async-timeout-3.0.1 attrs-21.2.0 auth0-python-3.6.1 autobahn-21.3.1 awesome-slugify-1.6.5 billiard-3.5.0.5 celery-4.2.1 cffi-1.14.6 channels-3.0.3 channels-redis-3.2.0 chardet-4.0.0 confluent-kafka-1.7.0 constantly-15.1.0 coreapi-2.3.3 coreschema-0.0.4 cryptography-3.4.6 daphne-3.0.1 defusedxml-0.7.1 django-3.2.5 django-allauth-0.34.0 django-celery-results-1.0.4 django-cors-headers-3.5.0 django-crispy-forms-1.7.0 django-debug-toolbar-2.2.1 django-elasticsearch-dsl-7.1.4 django-elasticsearch-dsl-drf-0.20.9 django-environ-0.4.4 django-extensions-2.2.9 django-model-utils-3.0.0 django-nine-0.2.4 django-redis-4.12.1 djangorestframework-3.12.2 djangorestframework-jwt-1.11.0 ecdsa-0.17.0 elasticsearch-7.13.4 elasticsearch-dsl-7.0.0 future-0.18.2 hiredis-2.0.0 hvac-0.10.1 hyperlink-21.0.0 idna-2.10 incremental-21.3.0 itypes-1.2.0 jinja2-3.0.1 kombu-4.3.0 msgpack-1.0.2 newrelic-6.4.4.161 oauthlib-3.1.1 psycopg2-2.8.5 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pyopenssl-20.0.1 python-dateutil-2.8.2 python-jose-3.0.1 python3-openid-3.2.0 pytz-2017.3 redis-3.5.3 regex-2021.7.6 requests-2.25.1 requests-oauthlib-1.3.0 rsa-4.7.2 service-identity-21.1.0 six-1.15.0 social-auth-app-django-3.1.0 social-auth-core-4.0.2 sorl-thumbnail-12.7.0 sorl-thumbnail-serializer-field-0.2.1 sqlparse-0.4.1 twisted-21.7.0 txaio-21.2.1 typing-extensions-3.10.0.0 uritemplate-3.0.1 urllib3-1.26.6 vine-1.3.0 wheel-0.30.0 zope.interface-5.4.0

real    0m10.434s
user    0m7.939s
sys     0m0.632s

I tried reproducing your issue on Linux with Python 3.9.6 and Pip 21.2.1, in my testing it took less than 11 seconds to install the requirements you provided:

Now try without caching. :) Like in a docker container.

I tried reproducing your issue on Linux with Python 3.9.6 and Pip 21.2.1, in my testing it took less than 11 seconds to install the requirements you provided:

Now try without caching. :) Like in a docker container.

I don't think being in the cache affects the path pip takes to trying to resolve dependencies, it certainly didn't seem to in the benchmarks above, but perhaps the devs can gives some advise on this. However one thing I did notice about my environment is I already had certfi preinstalled.

So I ran it again with the flags "--no-cace-dir" to prevent using the cache, and "--force-reinstall" to prevent it relying on any existing packages, it completed in less than 23 seconds, you can see it has to do a small amount of backtracking like on social_auth_core, but I don't think this is the performance issue you are reporting:

$ time python -m pip install --force-reinstall --no-cache-dir -r req.txt
Collecting wheel==0.30.0
  Downloading wheel-0.30.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 3.1 MB/s
Collecting django<3.3,>=3.2
  Downloading Django-3.2.5-py3-none-any.whl (7.9 MB)
     |████████████████████████████████| 7.9 MB 5.9 MB/s
Collecting six==1.15.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting django-environ==0.4.4
  Downloading django_environ-0.4.4-py2.py3-none-any.whl (24 kB)
Collecting django-crispy-forms==1.7.0
  Downloading django_crispy_forms-1.7.0-py2.py3-none-any.whl (104 kB)
     |████████████████████████████████| 104 kB 4.1 MB/s
Collecting django-model-utils==3.0.0
  Downloading django-model-utils-3.0.0.tar.gz (22 kB)
Collecting Pillow==8.2.0
  Downloading Pillow-8.2.0-cp39-cp39-manylinux1_x86_64.whl (3.0 MB)
     |████████████████████████████████| 3.0 MB 4.3 MB/s
Collecting argon2-cffi==20.1.0
  Downloading argon2_cffi-20.1.0-cp35-abi3-manylinux1_x86_64.whl (97 kB)
     |████████████████████████████████| 97 kB 7.0 MB/s
Collecting django-allauth==0.34.0
  Downloading django-allauth-0.34.0.tar.gz (475 kB)
     |████████████████████████████████| 475 kB 5.0 MB/s
Collecting psycopg2==2.8.5
  Downloading psycopg2-2.8.5.tar.gz (380 kB)
     |████████████████████████████████| 380 kB 5.3 MB/s
Collecting awesome-slugify==1.6.5
  Downloading awesome-slugify-1.6.5.tar.gz (8.4 kB)
Collecting pytz==2017.3
  Downloading pytz-2017.3-py2.py3-none-any.whl (511 kB)
     |████████████████████████████████| 511 kB 5.5 MB/s
Collecting django-redis==4.12.1
  Downloading django_redis-4.12.1-py3-none-any.whl (19 kB)
Collecting redis==3.5.3
  Downloading redis-3.5.3-py2.py3-none-any.whl (72 kB)
     |████████████████████████████████| 72 kB 6.6 MB/s
Collecting django-cors-headers==3.5.0
  Downloading django_cors_headers-3.5.0-py3-none-any.whl (11 kB)
Collecting djangorestframework==3.12.2
  Downloading djangorestframework-3.12.2-py3-none-any.whl (957 kB)
     |████████████████████████████████| 957 kB 7.8 MB/s
Collecting djangorestframework-jwt==1.11.0
  Downloading djangorestframework_jwt-1.11.0-py2.py3-none-any.whl (13 kB)
Collecting requests==2.25.1
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 5.3 MB/s
Collecting newrelic
  Downloading newrelic-6.6.0.162-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (705 kB)
     |████████████████████████████████| 705 kB 6.4 MB/s
Collecting cryptography==3.4.6
  Downloading cryptography-3.4.6-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
     |████████████████████████████████| 3.2 MB 3.9 MB/s
Collecting kombu==4.3.0
  Downloading kombu-4.3.0-py2.py3-none-any.whl (183 kB)
     |████████████████████████████████| 183 kB 7.5 MB/s
Collecting celery==4.2.1
  Downloading celery-4.2.1-py2.py3-none-any.whl (401 kB)
     |████████████████████████████████| 401 kB 4.9 MB/s
Collecting django-celery-results==1.0.4
  Downloading django_celery_results-1.0.4-py2.py3-none-any.whl (16 kB)
Collecting social-auth-app-django==3.1.0
  Downloading social_auth_app_django-3.1.0-py3-none-any.whl (23 kB)
Collecting python-jose==3.0.1
  Downloading python_jose-3.0.1-py2.py3-none-any.whl (25 kB)
Collecting elasticsearch_dsl==7.0.0
  Downloading elasticsearch_dsl-7.0.0-py2.py3-none-any.whl (48 kB)
     |████████████████████████████████| 48 kB 30.1 MB/s
Collecting django-elasticsearch-dsl==7.1.4
  Downloading django_elasticsearch_dsl-7.1.4-py2.py3-none-any.whl (19 kB)
Collecting django-elasticsearch-dsl-drf==0.20.9
  Downloading django_elasticsearch_dsl_drf-0.20.9-py2.py3-none-any.whl (146 kB)
     |████████████████████████████████| 146 kB 3.9 MB/s
Collecting channels==3.0.3
  Downloading channels-3.0.3-py3-none-any.whl (38 kB)
Collecting channels_redis==3.2.0
  Downloading channels_redis-3.2.0-py2.py3-none-any.whl (14 kB)
Collecting daphne==3.0.1
  Downloading daphne-3.0.1-py3-none-any.whl (26 kB)
Collecting sorl-thumbnail==12.7.0
  Downloading sorl_thumbnail-12.7.0-py3-none-any.whl (41 kB)
     |████████████████████████████████| 41 kB 3.7 MB/s
Collecting sorl-thumbnail-serializer-field==0.2.1
  Downloading sorl_thumbnail_serializer_field-0.2.1-py2.py3-none-any.whl (14 kB)
Collecting auth0-python==3.6.1
  Downloading auth0_python-3.6.1-py2.py3-none-any.whl (66 kB)
     |████████████████████████████████| 66 kB 8.0 MB/s
Collecting coreapi==2.3.3
  Downloading coreapi-2.3.3-py2.py3-none-any.whl (25 kB)
Collecting hvac==0.10.1
  Downloading hvac-0.10.1-py2.py3-none-any.whl (119 kB)
     |████████████████████████████████| 119 kB 4.1 MB/s
Collecting django-debug-toolbar==2.2.1
  Downloading django_debug_toolbar-2.2.1-py3-none-any.whl (200 kB)
     |████████████████████████████████| 200 kB 3.9 MB/s
Collecting django-extensions==2.2.9
  Downloading django_extensions-2.2.9-py2.py3-none-any.whl (217 kB)
     |████████████████████████████████| 217 kB 5.3 MB/s
Collecting confluent-kafka<1.8,>=1.7
  Downloading confluent_kafka-1.7.0-cp39-cp39-manylinux2010_x86_64.whl (2.7 MB)
     |████████████████████████████████| 2.7 MB 3.2 MB/s
Collecting cffi>=1.0.0
  Downloading cffi-1.14.6-cp39-cp39-manylinux1_x86_64.whl (405 kB)
     |████████████████████████████████| 405 kB 6.3 MB/s
Collecting python3-openid>=3.0.8
  Downloading python3_openid-3.2.0-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 5.1 MB/s
Collecting requests-oauthlib>=0.3.0
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting regex
  Downloading regex-2021.7.6-cp39-cp39-manylinux2014_x86_64.whl (733 kB)
     |████████████████████████████████| 733 kB 4.0 MB/s
Collecting Unidecode<0.05,>=0.04.14
  Downloading Unidecode-0.04.21-py2.py3-none-any.whl (228 kB)
     |████████████████████████████████| 228 kB 5.3 MB/s
Collecting PyJWT<2.0.0,>=1.5.2
  Downloading PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 6.7 MB/s
Collecting certifi>=2017.4.17
  Downloading certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
     |████████████████████████████████| 145 kB 4.6 MB/s
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 5.9 MB/s
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
     |████████████████████████████████| 138 kB 4.6 MB/s
Collecting amqp<3.0,>=2.4.0
  Downloading amqp-2.6.1-py2.py3-none-any.whl (48 kB)
     |████████████████████████████████| 48 kB 9.7 MB/s
Collecting billiard<3.6.0,>=3.5.0.2
  Downloading billiard-3.5.0.5.tar.gz (150 kB)
     |████████████████████████████████| 150 kB 5.4 MB/s
Collecting social-auth-core>=1.2.0
  Downloading social_auth_core-4.1.0-py3-none-any.whl (333 kB)
     |████████████████████████████████| 333 kB 6.0 MB/s
Collecting future<1.0
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 7.8 MB/s
Collecting rsa
  Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting ecdsa<1.0
  Downloading ecdsa-0.17.0-py2.py3-none-any.whl (119 kB)
     |████████████████████████████████| 119 kB 6.1 MB/s
Collecting elasticsearch<8.0.0,>=7.0.0
  Downloading elasticsearch-7.13.4-py2.py3-none-any.whl (356 kB)
     |████████████████████████████████| 356 kB 5.5 MB/s
Collecting python-dateutil
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB 6.7 MB/s
Collecting django-nine>=0.2
  Downloading django_nine-0.2.4-py2.py3-none-any.whl (25 kB)
Collecting asgiref<4,>=3.2.10
  Downloading asgiref-3.4.1-py3-none-any.whl (25 kB)
Collecting msgpack~=1.0
  Downloading msgpack-1.0.2-cp39-cp39-manylinux1_x86_64.whl (294 kB)
     |████████████████████████████████| 294 kB 5.4 MB/s
Collecting aioredis~=1.0
  Downloading aioredis-1.3.1-py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 6.8 MB/s
Collecting autobahn>=0.18
  Downloading autobahn-21.3.1-py2.py3-none-any.whl (495 kB)
     |████████████████████████████████| 495 kB 6.3 MB/s
Collecting twisted[tls]>=18.7
  Downloading Twisted-21.7.0-py3-none-any.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 4.4 MB/s
Collecting coreschema
  Downloading coreschema-0.0.4.tar.gz (10 kB)
Collecting itypes
  Downloading itypes-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting uritemplate
  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting sqlparse>=0.2.0
  Downloading sqlparse-0.4.1-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 3.6 MB/s
Collecting hiredis
  Downloading hiredis-2.0.0-cp39-cp39-manylinux2010_x86_64.whl (85 kB)
     |████████████████████████████████| 85 kB 8.5 MB/s
Collecting async-timeout
  Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting vine<5.0.0a1,>=1.1.3
  Downloading vine-1.3.0-py2.py3-none-any.whl (14 kB)
Collecting txaio>=21.2.1
  Downloading txaio-21.2.1-py2.py3-none-any.whl (30 kB)
Collecting hyperlink>=21.0.0
  Downloading hyperlink-21.0.0-py2.py3-none-any.whl (74 kB)
     |████████████████████████████████| 74 kB 8.5 MB/s
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 6.3 MB/s
Collecting defusedxml
  Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting oauthlib>=3.0.0
  Downloading oauthlib-3.1.1-py2.py3-none-any.whl (146 kB)
     |████████████████████████████████| 146 kB 13.4 MB/s
Collecting social-auth-core>=1.2.0
  Downloading social_auth_core-4.0.3-py3-none-any.whl (328 kB)
     |████████████████████████████████| 328 kB 4.7 MB/s
  Downloading social_auth_core-4.0.2-py3-none-any.whl (330 kB)
     |████████████████████████████████| 330 kB 4.2 MB/s
Collecting constantly>=15.1
  Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting incremental>=21.3.0
  Downloading incremental-21.3.0-py2.py3-none-any.whl (15 kB)
Collecting attrs>=19.2.0
  Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 7.2 MB/s
Collecting typing-extensions>=3.6.5
  Downloading typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Collecting zope.interface>=4.4.2
  Downloading zope.interface-5.4.0-cp39-cp39-manylinux2010_x86_64.whl (255 kB)
     |████████████████████████████████| 255 kB 5.0 MB/s
Collecting Automat>=0.8.0
  Downloading Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting service-identity>=18.1.0
  Downloading service_identity-21.1.0-py2.py3-none-any.whl (12 kB)
Collecting pyopenssl>=16.0.0
  Downloading pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 7.0 MB/s
Collecting pyasn1-modules
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 3.8 MB/s
Collecting pyasn1
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 6.7 MB/s
Collecting setuptools
  Downloading setuptools-57.4.0-py3-none-any.whl (819 kB)
     |████████████████████████████████| 819 kB 4.9 MB/s
Collecting jinja2
  Downloading Jinja2-3.0.1-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 5.5 MB/s
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1-cp39-cp39-manylinux2010_x86_64.whl (30 kB)
Building wheels for collected packages: django-model-utils, django-allauth, psycopg2, awesome-slugify, billiard, future, coreschema
  Building wheel for django-model-utils (setup.py) ... done
  Created wheel for django-model-utils: filename=django_model_utils-3.0.0-py2.py3-none-any.whl size=20835 sha256=29a32700baaf3d9c30f74a74b668e54552722e76d7c9ce3d6e09ef37ac782724
  Stored in directory: /tmp/pip-ephem-wheel-cache-6w1h0ajw/wheels/9c/e7/93/b14c55e22df6b633f3e93a79c42ba03f628b8897302520f4f8
  Building wheel for django-allauth (setup.py) ... done
  Created wheel for django-allauth: filename=django_allauth-0.34.0-py3-none-any.whl size=744993 sha256=3665da06b2c8783d3fcfaaa4260b45eb8a3783c32ad3cf0efbcc6d434f9aac71
  Stored in directory: /tmp/pip-ephem-wheel-cache-6w1h0ajw/wheels/97/30/e2/e59276ad2eef7382dd1b82938258149fbd3b4483c4d6999980
  Building wheel for psycopg2 (setup.py) ... done
  Created wheel for psycopg2: filename=psycopg2-2.8.5-cp39-cp39-linux_x86_64.whl size=154496 sha256=f0f992f6a6637c263c0f476133a7448fce4c239594eda893070b7dfce964d1b1
  Stored in directory: /tmp/pip-ephem-wheel-cache-6w1h0ajw/wheels/c2/17/82/f619fa1d1a361445c4ff28634f734936f2d54891c79840b345
  Building wheel for awesome-slugify (setup.py) ... done
  Created wheel for awesome-slugify: filename=awesome_slugify-1.6.5-py3-none-any.whl size=8351 sha256=116a34c89ea1694859993e9a1864858c96a13d47cc1f3c25d2dcad171f00a71e
  Stored in directory: /tmp/pip-ephem-wheel-cache-6w1h0ajw/wheels/fa/1f/c1/1ee7ffdaf5c2499aa4e87a520e99ae44a2888d540811c7dc30
  Building wheel for billiard (setup.py) ... done
  Created wheel for billiard: filename=billiard-3.5.0.5-py3-none-any.whl size=87880 sha256=782d709e7135440752e0294aab022e0defdda7c9d402a0b8c9d2472b64c2d00a
  Stored in directory: /tmp/pip-ephem-wheel-cache-6w1h0ajw/wheels/cc/a8/5d/8fc2e8e4255e5e33e4b3a26ade7a7385e67b4992a8b7e697f9
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=0b6b7016fdb04835d58ebae8f9fdc53438e20ffa0a4ef0626719e783b99ba9a1
  Stored in directory: /tmp/pip-ephem-wheel-cache-6w1h0ajw/wheels/2f/a0/d3/4030d9f80e6b3be787f19fc911b8e7aa462986a40ab1e4bb94
  Building wheel for coreschema (setup.py) ... done
  Created wheel for coreschema: filename=coreschema-0.0.4-py3-none-any.whl size=15031 sha256=cd7d400607ff19a4fa14307660caf115eda9a2f4ea2b7915ca944b5a124b7db3
  Stored in directory: /tmp/pip-ephem-wheel-cache-6w1h0ajw/wheels/14/d2/2e/b54b3aa0f4d8a9f7bd628dc0ac5edb277e5ce42e45bd89dacd
Successfully built django-model-utils django-allauth psycopg2 awesome-slugify billiard future coreschema
Installing collected packages: pycparser, six, setuptools, pyasn1, idna, cffi, attrs, zope.interface, typing-extensions, pyasn1-modules, incremental, hyperlink, cryptography, constantly, Automat, vine, urllib3, txaio, twisted, service-identity, pyopenssl, chardet, certifi, sqlparse, requests, pytz, python-dateutil, oauthlib, MarkupSafe, elasticsearch, defusedxml, autobahn, asgiref, amqp, requests-oauthlib, python3-openid, PyJWT, kombu, jinja2, hiredis, elasticsearch-dsl, django, daphne, billiard, async-timeout, uritemplate, Unidecode, sorl-thumbnail, social-auth-core, rsa, regex, redis, msgpack, itypes, future, ecdsa, djangorestframework, django-nine, django-elasticsearch-dsl, coreschema, channels, celery, aioredis, wheel, sorl-thumbnail-serializer-field, social-auth-app-django, python-jose, psycopg2, Pillow, newrelic, hvac, djangorestframework-jwt, django-redis, django-model-utils, django-extensions, django-environ, django-elasticsearch-dsl-drf, django-debug-toolbar, django-crispy-forms, django-cors-headers, django-celery-results, django-allauth, coreapi, confluent-kafka, channels-redis, awesome-slugify, auth0-python, argon2-cffi
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.6.0.post20210108
    Uninstalling setuptools-49.6.0.post20210108:
      Successfully uninstalled setuptools-49.6.0.post20210108
  Attempting uninstall: certifi
    Found existing installation: certifi 2021.5.30
    Uninstalling certifi-2021.5.30:
      Successfully uninstalled certifi-2021.5.30
  Attempting uninstall: wheel
    Found existing installation: wheel 0.36.2
    Uninstalling wheel-0.36.2:
      Successfully uninstalled wheel-0.36.2
Successfully installed Automat-20.2.0 MarkupSafe-2.0.1 Pillow-8.2.0 PyJWT-1.7.1 Unidecode-0.4.21 aioredis-1.3.1 amqp-2.6.1 argon2-cffi-20.1.0 asgiref-3.4.1 async-timeout-3.0.1 attrs-21.2.0 auth0-python-3.6.1 autobahn-21.3.1 awesome-slugify-1.6.5 billiard-3.5.0.5 celery-4.2.1 certifi-2021.5.30 cffi-1.14.6 channels-3.0.3 channels-redis-3.2.0 chardet-4.0.0 confluent-kafka-1.7.0 constantly-15.1.0 coreapi-2.3.3 coreschema-0.0.4 cryptography-3.4.6 daphne-3.0.1 defusedxml-0.7.1 django-3.2.5 django-allauth-0.34.0 django-celery-results-1.0.4 django-cors-headers-3.5.0 django-crispy-forms-1.7.0 django-debug-toolbar-2.2.1 django-elasticsearch-dsl-7.1.4 django-elasticsearch-dsl-drf-0.20.9 django-environ-0.4.4 django-extensions-2.2.9 django-model-utils-3.0.0 django-nine-0.2.4 django-redis-4.12.1 djangorestframework-3.12.2 djangorestframework-jwt-1.11.0 ecdsa-0.17.0 elasticsearch-7.13.4 elasticsearch-dsl-7.0.0 future-0.18.2 hiredis-2.0.0 hvac-0.10.1 hyperlink-21.0.0 idna-2.10 incremental-21.3.0 itypes-1.2.0 jinja2-3.0.1 kombu-4.3.0 msgpack-1.0.2 newrelic-6.6.0.162 oauthlib-3.1.1 psycopg2-2.8.5 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pyopenssl-20.0.1 python-dateutil-2.8.2 python-jose-3.0.1 python3-openid-3.2.0 pytz-2017.3 redis-3.5.3 regex-2021.7.6 requests-2.25.1 requests-oauthlib-1.3.0 rsa-4.7.2 service-identity-21.1.0 setuptools-57.4.0 six-1.15.0 social-auth-app-django-3.1.0 social-auth-core-4.0.2 sorl-thumbnail-12.7.0 sorl-thumbnail-serializer-field-0.2.1 sqlparse-0.4.1 twisted-21.7.0 txaio-21.2.1 typing-extensions-3.10.0.0 uritemplate-3.0.1 urllib3-1.26.6 vine-1.3.0 wheel-0.30.0 zope.interface-5.4.0

real    0m22.268s
user    0m12.682s
sys     0m1.179s

Here's the dockerfile, that when combined with the requirements.txt above produced my results:

FROM python:3-slim

ENV PYTHONUNBUFFERED 1

RUN apt-get update \
    && apt-get install -y swig libssl-dev dpkg-dev netcat libpq-dev \
    && apt-get clean autoclean \
    && apt-get autoremove --yes \
    && rm -rf /var/lib/{apt,dpkg,cache,log}/ \
    && rm -rf /var/lib/apt/lists/*

RUN python -m pip install -U --force-reinstall pip

COPY requirements.txt /requirements.txt

RUN pip install --no-cache-dir -r /requirements.txt

Here's the dockerfile, that when combined with the requirements.txt above produced my results:

I built the Dockerfile and it took 24.1 seconds to run the pip install --no-cache-dir -r /requirements.txt step:

~/docker_file_test$ time docker build --no-cache -t pip_dep_test .
[+] Building 53.6s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 476B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/python:3-slim                                                   6.0s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 880B                                                                                  0.0s
 => CACHED [1/5] FROM docker.io/library/python:3-slim@sha256:c5f60863db103c951595f110def9244c1e09efe9e8d072cfac3d  0.0s
 => [2/5] RUN apt-get update     && apt-get install -y swig libssl-dev dpkg-dev netcat libpq-dev     && apt-get   19.9s
 => [3/5] RUN python -m pip install -U --force-reinstall pip                                                       2.5s
 => [4/5] COPY requirements.txt /requirements.txt                                                                  0.0s
 => [5/5] RUN pip install --no-cache-dir -r /requirements.txt                                                     24.1s
 => exporting to image                                                                                             1.0s
 => => exporting layers                                                                                            0.9s
 => => writing image sha256:047500b50045618ed0202c3269686820a0b6d3d6a982c97d68335120798125c0                       0.0s
 => => naming to docker.io/library/pip_dep_test                                                                    0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

real    0m53.838s
user    0m0.283s
sys     0m0.142s

I haven’t been able to take a good look into this, but a couple of notes:

I’m going to release 21.2.2 for some other bugs in 21.2.1 and this will not be fixed in it (I’ll push the milestone back).
One of the advantages of the new resolve logic is it’s possible to tweak its behaviour by reordering your dependencies, and/or adding some of the dependencies to the requirements file. A lot of the performance depends on the ordering of things, so you can work out the best ordering for your project if the resolver can’t (and it never will be able to find the best route 100% of the time, all we can do is make the worst case scenario happen less and less tasxing them it happens).

2. One of the advantages of the new resolve logic is it’s possible to tweak its behaviour _by reordering your dependencies_, and/or adding some of the dependencies to the requirements file. A lot of the performance depends on the ordering of things, so you can work out the best ordering for your project if the resolver can’t (and it never will be able to find the best route 100% of the time, all we can do is make the worst case scenario happen less and less tasxing them it happens).

Is there some documentation on how the ordering works here? It's not obvious to those of us who don't understand pips resolution engine.

Based on this comment I have looked at OPs test case and found an issue with it. In the first line of the requirements file they specify installing their own package:

-e .

But this package has a install requirements in the setup.py that do not align with the install requirements in their requirements file:

    "boto3>=1.9.201",
    "botocore>=1.12.201"

Therefore I think the amalgamated requirement list pip is trying to resolve ends up looking like this:

.
boto3>=1.9.201
botocore>=1.12.201
pytest-cov; python_version >= '3.6'
coverage==4.5.4
boto>=2.45.0

OP can fix their issue in one of 2 ways (actually preferably both):

Remove the "-e ." in their requirements text. I don't think it's standard practice to locally refer to a package in the requirements text? You can move installing the local package out to a separate command in your CI tool after you've installed your requirements. If you really want to include it moving it to the end of the requirements file should also fix the issue.
Update the setup.py and requirements files so they both include all the requirements, listing the boto packages at the end in both.

Applying either or both will fix OPs issue in my testing, perhaps they can confirm? No one else on this thread has produced a reproducible issue.

That said IMO I think Pip should do more to highlight 1) Ordering is important, 2) What is the order is received from the amalgamated requirements it builds up (in OPs example the requirements end up being split across 3 files), 3) Giving more options for pip to not even attempt to backtrack so deeply down a single requirement (e.g. max-backtrack-depth-for-single-requirement or something).

Is there some documentation on how the ordering works here? It's not obvious to those of us who don't understand pips resolution engine.

I don't think there is. To be honest, it's not clear to (at least some of) those of us who do understand pip's resolution engine 🙁 @uranusjr may be able to give some hints, though.

Based on this comment I have looked at OPs test case and found an issue with it.

Nice work!

That said IMO I think Pip should do more to highlight 1) Ordering is important, 2) What is the order is received from the amalgamated requirements it builds up (in OPs example the requirements end up being split across 3 files), 3) Giving more options for pip to not even attempt to backtrack so deeply down a single requirement (e.g. max-backtrack-depth-for-single-requirement or something).

Agreed, these would be potentially useful.

A documentation PR would be appreciated here. Sadly, I think this is something that needs user contribution more than core developer time. It's quite hard when you're close to the implementation to know what's "obvious" and what isn't, and to express the key points in a way that an end user would be able to use. So if someone from the user side of the equation could offer a starting point in the form of a PR that tries to express the information, the pip devs can probably correct any factual issues. If that results in something that is still useful (there's a risk that we end up with so many caveats that it's not actually helpful), then we can add it to the docs.
When we call the resolver, we have a set of requirements (around here) - we could print that, but I don't think it's as useful as you'd hope - in the OP's case, read reqs.txt and insert req-tests.txt at the point where it's referenced. That's all (I think 😉). We only dive into the requirements of the project itself (the ones in setup.py) when we start trying to install the project and its requirements.
Options like this probably need a PR to resolvelib, and then a change to pip to use that. As this would very much be a resolvelib change, creating an "artificial" example that builds a silly-deep resolution tree, and runs it through resolvelib to demonstrate the issue would be a great start - the resolvelib test suite gives good examples of this style of test.

Maybe another useful resource would be some sort of "user community help" forum, where people working on projects with complex and tricky dependency trees (ahem boto users 😉) could help each other by sharing experience, helping to diagnose issues, and preparing reproducers for intractable issues? The pip devs would probably be interested in participating to advise, too. I'm not sure the issue tracker is necessarily the most helpful forum (we're very much focused here on verifying if there's a pip bug, and identifying changes to pip that might be needed, not on debugging user configurations).

But this package has a install requirements in the setup.py that do not align with the install requirements in their requirements file

What do you mean by this @notatallshaw? Is there an interdependency between these requirements that would make resolving this list impossible?

1. Remove the "-e ." in their requirements text. I don't think it's standard practice to locally refer to a package in the requirements text?

It's part of the documentation as a valid option: https://pip.pypa.io/en/stable/cli/pip_install/#requirements-file-format I don't know whether it's standard practice, but if this functionality was deprecated in the latest release, I'd assume it would be documented somewhere.

2. Update the `setup.py` and requirements files so they both include all the requirements, listing the `boto` packages at the end in both.
That brings the pain of having to maintain two sets of requirements, which is exactly what our setup avoids (unusual as it may be).

Applying either or both will fix OPs issue

They might provide a workaround, yes. It's not a fix. Same goes for the ordering. I'm sure that helps the resolver, and it would be very useful for large/complex/tricky dependency trees. But we're talking about 5 direct dependencies here, where the latest/required versions all play nice together.

To reiterate - this project worked absolutely fine in the previous release. To me, that's a pretty big hint that it's a valid setup.

What do you mean by this @notatallshaw? Is there an interdependency between these requirements that would make resolving this list impossible?

It is standard practice to be able to run python -m pip install . and for the relevant dependencies to be fully installed, in your case this does not work as you have split your requirements up in an odd way that makes your problem difficult to debug (as demonstrated by the long conversation above).

It's not impossible to resolve, in fact I benchmark above how long it takes to install.

But we're talking about 5 direct dependencies here, where the latest/required versions all play nice together.

To be clear we're talking about a dependency graph with thousands of nodes and requirements, and possible paths to take across this graph a combinatorial of that. I'm not part of the pip team so I can't say what constitutes a fix but you're drastically oversimplifying the problem here. If you want better performance specify better requirements.

It's part of the documentation as a valid option: https://pip.pypa.io/en/stable/cli/pip_install/#requirements-file-format I don't know whether it's standard practice, but if this functionality was deprecated in the latest release, I'd assume it would be documented somewhere.

Just because you can do something and it's not deprecated doesn't make it a good idea

To reiterate - this project worked absolutely fine in the previous release. To me, that's a pretty big hint that it's a valid setup.

It is a valid setup, as long as you don't care about performance, I can install your set-up perfectly fine on my machine with pip 21.2.1, it takes 15-30 mins on my machine depending if the packages are cached or not. Just because you were lucky before on how pip went through this large dependency graph doesn't mean you were doing the right thing. It also doesn't mean pip is being very helpful here, as I have stated above.

@uranusjr I took a deeper look at this example, I think you might find my findings interesting, there might be a bug here? (or at least an optimization?)

First I reduced it down a more minimal reproducible example:

In the requirements file:

.
pytest-cov
coverage==4.5.4

In setup.py:

install_requires = [
    "boto3>=1.9.201",
]

Observations:

pytest-cov has a requirement on coverage[toml] and needs to backtrack to an older version to not conflict with coverage==4.5.4
Instead pip keeps trying to find older and older versions of botocore, which as best as I can tell do not conflict with the coverage==4.5.4, though I don't yet understand how pip is picking new candidates to download

If I just create a requirements file with these 3 requirements it resolves quite quickly and it doesn't attempt to backtrack on boto3 or botocore:

boto3>=1.9.201
pytest-cov
coverage==4.5.4

My initial thoughts were that because boto3 is in the install_requires then botocore has a higher implied depth than pytest-cov but I don't see that when I print out the results of _get_preference, and it also doesn't make sense to me why new botocore candidates are being picked when I don't see any conflicts? The only conflicts I see are between pytest-covs requirement on coverage[toml] and coverage==4.5.4.

Is this perhaps something where resolver isn't understanding where the conflict is because it comes from an "extras" requirement (i.e [toml])? But then why does the second example of just putting all 3 requirements in to the requirements file work?

One thing that I don't think is commonly known¹ and which might be useful for people hitting problems with long resolution times involving boto, is that if you have a constraint file containing version limits for a project, those limits are applied when pip fetches the list of versions from the package index, i.e., before the resolution and backtracking step. Conversely, version limits that are in a requirements file and/or individual project dependencies, are only applied when the dependency is first encountered (which may be after the version list is fetched).

So as a rule of thumb:

If you want to force pip to only consider versions of a package newer that a given version, add pkg>=version to a constraints file and supply that (via the --constraint <file> option) to pip. This can be in addition to any existing requirements in requirements.txt or in project dependencies.

So, for example, in the case here, try adding

boto3>=1.9.201
botocore>=1.12.201

to a constraints.txt file, and see if that helps.

¹ By which I mean that I thought it would be a useful option to add, went and looked at the code and discovered that I'd implemented it when I first added constraint support to the new resolver, but had forgotten about it 🙂

To add to this converation

Less than 0.8% of packages on PyPI have more than 100 releases, so in principle it's entirely reasonable for the trade-offs that pip makes to favour projects with fewer releases. Unfortunately, projects like botocore are too popular to be considered "outliers" in a general sense. I wonder if we could prioritise projects with fewer releases over those with more? Or is that something we've already tried?

So while true most packages only have a few release, unfortunately looking at the top 10 packages on PyPi 50% have over 100 releases: botocore, awscli, boto3, requests, setuptools.

In the past (not sure if it's still true) setuptools has been identified in the resolver as a special case, I think it was to deprioritize checking for older versions? IMO I think it makes sense to add the other 4 packages listed here to that special case, but at the least add botocore, awcli, and boto3 as it looks like all 3 packages release daily.

What makes the pip resolver interesting about it's dependency resolution vs. other dependency resolver like conda is that exploring the dependency graph is a non-trivial cost. And it makes intuitive sense that packages with very frequent release (e.g. daily) will not change their dependency requirements every release and therefore trying to backtrack through those packages is going to be highly costly (this is my intuition and unsubstantiated by any data, I'll have a think if there's anyway I could empirically collect this information). But as @uranusjr points out this information currently isn't propagated to the pip resolver. So maybe it makes sense to special case very popular very commonly released packages?

My 2 cents.

To add to this discussion. There is an active thread of work on awscli, which appears to be concluding that metadata isn't being made available because that project is uploading their built artifacts to pypi using an older client, so the dependency information isn't available and necessitates a download.

awscli, boto3, and botocore are all AWS produced packages, which increases the likelihood they're deploying built objects using the same old client. Resolving that wouldn't make old packages immediately correct, but it could make this less of an issue going forward for those giant, popular packages.

metadata isn't being made available because that project is uploading their built artifacts to pypi using an older client, so the dependency information isn't available and necessitates a download.

Unfortunately, it's not about a single package here. Pip doesn't use the PyPI JSON interface to get metadata, because it's unreliable in general - precisely because it's supplied by the upload client and not read from the package file itself. There's work ongoing to allow PyPI (and other indexes conforming to the spec) to provide reliable metadata, but it won't be via the JSON API.

One thing that I don't think is commonly known¹ and which might be useful for people hitting problems with long resolution times involving boto, is that if you have a constraint file containing version limits for a project, those limits are applied when pip fetches the list of versions from the package index, i.e., before the resolution and backtracking step. Conversely, version limits that are in a requirements file and/or individual project dependencies, are only applied when the dependency is first encountered (which may be after the version list is fetched).

So as a rule of thumb:

If you want to force pip to only consider versions of a package newer that a given version, add pkg>=version to a constraints file and supply that (via the --constraint <file> option) to pip. This can be in addition to any existing requirements in requirements.txt or in project dependencies.

So, for example, in the case here, try adding
boto3>=1.9.201
botocore>=1.12.201
to a constraints.txt file, and see if that helps.

Adding a constraints file doesn't help in this case (just tested).

This particular issue goes away when you specify all the requirements in the requirements file (or the setup.py file) as boto3 and botocore aren't causing any conflict with the other requirements.

As I posted above the conflict is happening between pytest-cov and coverage==4.5.4.

But because boto3 is listed only in the setup.py where as the other requirements are listed only in the requirements file it seems to be exclusively choosing boto3, and it's requirement packages, as candidates to download next even though they aren't causing any conflicts. I believe this not intended behavior and therefore is a bug in pip (or resolvelib) that should be addressed.

However because that bug is so specific to OPs use case I am inclined to think that other people posting here are having long resolution times for different reasons. But no one else has posted a reproducible example so I can't compare.

Here's a reproducible example: I'm using python 3.7.8 and poetry 1.1.7 to build this pyproject.toml.

[tool.poetry]
name = "someinternalname"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.7"
umap-learn = ">=0.3.10"
hdbscan = ">=0.8.15"
flake8 = "3.7.9"
pytest = ">=5.3.2"
presto-python-client = ">=0.7.0"
pyhive = ">=0.6.1"
argparse = ">=1.4.0"
pandas = ">=0.25.3"
boto3 = ">=1.10.46"
moto = ">=1.3.14"
hypothesis = ">=5.1.1"
apache-airflow = "1.10.9"
yapf = "0.29.0"
mypy = "^0.761"
statsd = "^3.3.0"
fabric = "^2.5.0"
psycopg2-binary = "^2.8.4"
sasl = "^0.2.1"
thrift-sasl = "^0.3.0"
pyarrow = "^0.15.1"
awscli = "1.20.3"
networkx = "^2.4"
coverage = "^5.5"
lxml = "^4.6.3"
great-expectations = "^0.13.25"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Dependency resolution using pip 21.1.1 takes 36 seconds, and in 21.2.X, it takes in the hundreds of seconds.

pypa / pip