Closed pabloa closed 2 years ago
Same issue happened to me today, when using python2.7 though:
Traceback (most recent call last):
File "/usr/local/bin/pip", line 7, in <module>
from pip._internal.cli.main import main
File "/usr/local/lib/python2.7/dist-packages/pip/_internal/cli/main.py", line 60
sys.stderr.write(f"ERROR: {exc}")
^
SyntaxError: invalid syntax
Your Python installation (2.7) is no longer supported by pip. This is not a bug. You can’t upgrade to pip 21.0.
Oops, the OP was actually describin a different issue. I’ll mark these comments as off-topic.
Yes. All I described happens in Python 3.7.x
This seems to be a case where the resolver is trying too hard to find a solution that doesn't exist. I think that's the case because of the warning at the end of the pip deptree that's been posted.
I think this can be bundled into #9187, since nothing about this is specific to 21.0 AFAICT.
It used to work with the old resolver. The current behavior is clearly a pip==21.0 issue. It does not happen with pip 20.x, for example, or older versions.
If pip tries to find a solution and it does not exist, it should print a warning and solves it like before or by doing an educated guess, or something. There are several projects like ours. They use/depend on packages with a lack of information. We still must deploy to production.
The old resolver “works” in the same sense that a random person assess situations faster than an expert. The rando guesses randomly, while the expert needs to actually analyse the situation. If the old resolver gets things right, you’re lucky. But that’s not a very useful baseline to evaluate performance.
@uranusjr I am not saying pip must use the old resolver. I am saying: If the new resolver cannot solve something because there are 1 or more bugs somewhere on the dependent packages, it should behave for that execution as the old resolver. Or at least pick something. And print a warning.
I recognize the new resolver is predictable and better. Still, there are scenarios like the one I reported where this resolver does not work. It was running for 3 hours and could not install all the dependencies!
Is it unreasonable to pick the best candidate package on those cases? The new resolver does not event report which package is unsolvable.
It has to be a better solution for this. And we are not the only ones having problems.
If picking one version of a package is not acceptable when the resolver cannot resolve, please suggest some alternative: print the package having issues? Restarting the installation using the old resolver?
Being unable to install a project with new versions of pip does not seem a solution.
I was trying to reproduce this example to find ways that pip resolver could be improved but it seems that is is no longer reproducible?
I used Python 3.7.3 and Pip 21.0.1 with this setup.py
to try to reproduce it:
from setuptools import setup
setup(
name="foo",
version="1.0",
install_requires=[
'spacy==2.2.3',
'sentence-transformers',
'sentencepiece==0.1.85',
'scispacy>=0.2.0',
'nltk',
'python-docx',
'langdetect',
'fuzzywuzzy==0.15.0',
'python-Levenshtein==0.12.0',
'annoy==1.16.3',
'pyDatalog',
'srsly==1.0.1',
'nest_asyncio==1.4.2']
)
And it successfully built:
(pip_test) PS test_pip> pip install -e .
Obtaining test_pip
Collecting spacy==2.2.3
Downloading spacy-2.2.3-cp37-cp37m-win_amd64.whl (9.7 MB)
|████████████████████████████████| 9.7 MB 3.3 MB/s
Collecting sentence-transformers
Downloading sentence-transformers-0.4.1.2.tar.gz (64 kB)
|████████████████████████████████| 64 kB 4.5 MB/s
Collecting sentencepiece==0.1.85
Downloading sentencepiece-0.1.85-cp37-cp37m-win_amd64.whl (1.2 MB)
|████████████████████████████████| 1.2 MB ...
Collecting scispacy>=0.2.0
Downloading scispacy-0.4.0-py3-none-any.whl (44 kB)
|████████████████████████████████| 44 kB ...
Collecting nltk
Downloading nltk-3.5.zip (1.4 MB)
|████████████████████████████████| 1.4 MB 6.4 MB/s
Collecting python-docx
Downloading python-docx-0.8.10.tar.gz (5.5 MB)
|████████████████████████████████| 5.5 MB 6.8 MB/s
Collecting langdetect
Downloading langdetect-1.0.8.tar.gz (981 kB)
|████████████████████████████████| 981 kB 6.4 MB/s
Collecting fuzzywuzzy==0.15.0
Downloading fuzzywuzzy-0.15.0-py2.py3-none-any.whl (28 kB)
Collecting python-Levenshtein==0.12.0
Using cached python-Levenshtein-0.12.0.tar.gz (48 kB)
Collecting annoy==1.16.3
Downloading annoy-1.16.3.tar.gz (644 kB)
|████████████████████████████████| 644 kB 6.4 MB/s
Collecting pyDatalog
Downloading pyDatalog-0.17.1.zip (304 kB)
|████████████████████████████████| 304 kB ...
Collecting srsly==1.0.1
Downloading srsly-1.0.1-cp37-cp37m-win_amd64.whl (179 kB)
|████████████████████████████████| 179 kB ...
Collecting nest_asyncio==1.4.2
Downloading nest_asyncio-1.4.2-py3-none-any.whl (5.3 kB)
Requirement already satisfied: setuptools in pip_test\lib\site-packages (from python-Levenshtein==0.12.0->foo==1.0) (52.0.0.post20210125)
Collecting preshed<3.1.0,>=3.0.2
Downloading preshed-3.0.5-cp37-cp37m-win_amd64.whl (108 kB)
|████████████████████████████████| 108 kB ...
Collecting blis<0.5.0,>=0.4.0
Downloading blis-0.4.1-cp37-cp37m-win_amd64.whl (5.0 MB)
|████████████████████████████████| 5.0 MB 6.8 MB/s
Collecting cymem<2.1.0,>=2.0.2
Downloading cymem-2.0.5-cp37-cp37m-win_amd64.whl (35 kB)
Collecting catalogue<1.1.0,>=0.0.7
Downloading catalogue-1.0.0-py2.py3-none-any.whl (7.7 kB)
Collecting thinc<7.4.0,>=7.3.0
Downloading thinc-7.3.1-cp37-cp37m-win_amd64.whl (2.0 MB)
|████████████████████████████████| 2.0 MB 6.8 MB/s
Collecting numpy>=1.15.0
Downloading numpy-1.20.1-cp37-cp37m-win_amd64.whl (13.6 MB)
|████████████████████████████████| 13.6 MB 3.3 MB/s
Collecting murmurhash<1.1.0,>=0.28.0
Downloading murmurhash-1.0.5-cp37-cp37m-win_amd64.whl (20 kB)
Collecting requests<3.0.0,>=2.13.0
Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting plac<1.2.0,>=0.9.6
Downloading plac-1.1.3-py2.py3-none-any.whl (20 kB)
Collecting wasabi<1.1.0,>=0.4.0
Downloading wasabi-0.8.2-py3-none-any.whl (23 kB)
Collecting importlib-metadata>=0.20
Downloading importlib_metadata-3.4.0-py3-none-any.whl (10 kB)
Collecting zipp>=0.5
Downloading zipp-3.4.0-py3-none-any.whl (5.2 kB)
Collecting typing-extensions>=3.6.4
Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
|████████████████████████████████| 137 kB ...
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: certifi>=2017.4.17 in pip_test\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy==2.2.3->foo==1.0) (2020.12.5)
Collecting scispacy>=0.2.0
Downloading scispacy-0.3.0-py3-none-any.whl (42 kB)
|████████████████████████████████| 42 kB 3.2 MB/s
Collecting nmslib>=1.7.3.6
Downloading nmslib-2.1.1-cp37-cp37m-win_amd64.whl (662 kB)
|████████████████████████████████| 662 kB ...
Collecting scispacy>=0.2.0
Downloading scispacy-0.2.5-py3-none-any.whl (38 kB)
Downloading scispacy-0.2.4.tar.gz (38 kB)
Collecting awscli
Downloading awscli-1.19.12.tar.gz (1.4 MB)
|████████████████████████████████| 1.4 MB 6.8 MB/s
Collecting conllu
Downloading conllu-4.4-py2.py3-none-any.whl (15 kB)
Collecting joblib
Downloading joblib-1.0.1-py3-none-any.whl (303 kB)
|████████████████████████████████| 303 kB ...
Collecting scikit-learn>=0.20.3
Downloading scikit_learn-0.24.1-cp37-cp37m-win_amd64.whl (6.8 MB)
|████████████████████████████████| 6.8 MB 6.4 MB/s
Collecting pysbd
Downloading pysbd-0.3.4-py3-none-any.whl (71 kB)
|████████████████████████████████| 71 kB ...
Collecting psutil
Downloading psutil-5.8.0-cp37-cp37m-win_amd64.whl (244 kB)
|████████████████████████████████| 244 kB ...
Collecting pybind11<2.6.2
Downloading pybind11-2.6.1-py2.py3-none-any.whl (188 kB)
|████████████████████████████████| 188 kB 6.8 MB/s
Collecting scipy>=0.19.1
Downloading scipy-1.6.1-cp37-cp37m-win_amd64.whl (32.6 MB)
|████████████████████████████████| 32.6 MB 6.4 MB/s
Collecting threadpoolctl>=2.0.0
Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)
Collecting tqdm<5.0.0,>=4.10.0
Downloading tqdm-4.57.0-py2.py3-none-any.whl (72 kB)
|████████████████████████████████| 72 kB 688 kB/s
Collecting botocore==1.20.12
Downloading botocore-1.20.12-py2.py3-none-any.whl (7.2 MB)
|████████████████████████████████| 7.2 MB 1.7 MB/s
Collecting docutils<0.16,>=0.10
Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
|████████████████████████████████| 547 kB ...
Collecting s3transfer<0.4.0,>=0.3.0
Downloading s3transfer-0.3.4-py2.py3-none-any.whl (69 kB)
|████████████████████████████████| 69 kB ...
Collecting PyYAML<5.5,>=3.10
Downloading PyYAML-5.4.1-cp37-cp37m-win_amd64.whl (210 kB)
|████████████████████████████████| 210 kB ...
Collecting colorama<0.4.4,>=0.2.5
Downloading colorama-0.4.3-py2.py3-none-any.whl (15 kB)
Collecting rsa<=4.5.0,>=3.1.2
Downloading rsa-4.5-py2.py3-none-any.whl (36 kB)
Collecting jmespath<1.0.0,>=0.7.1
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six>=1.5
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pyasn1>=0.1.3
Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting click
Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
|████████████████████████████████| 82 kB 483 kB/s
Collecting regex
Downloading regex-2020.11.13-cp37-cp37m-win_amd64.whl (269 kB)
|████████████████████████████████| 269 kB ...
Collecting lxml>=2.3.2
Downloading lxml-4.6.2-cp37-cp37m-win_amd64.whl (3.5 MB)
|████████████████████████████████| 3.5 MB 3.3 MB/s
Collecting transformers<5.0.0,>=3.1.0
Downloading transformers-4.3.2-py3-none-any.whl (1.8 MB)
|████████████████████████████████| 1.8 MB ...
Collecting torch>=1.6.0
Downloading torch-1.7.1-cp37-cp37m-win_amd64.whl (184.1 MB)
|████████████████████████████████| 184.1 MB 4.6 kB/s
Collecting sacremoses
Downloading sacremoses-0.0.43.tar.gz (883 kB)
|████████████████████████████████| 883 kB 6.4 MB/s
Collecting tokenizers<0.11,>=0.10.1
Downloading tokenizers-0.10.1-cp37-cp37m-win_amd64.whl (2.0 MB)
|████████████████████████████████| 2.0 MB 6.8 MB/s
Collecting filelock
Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting packaging
Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
|████████████████████████████████| 40 kB ...
Collecting pyparsing>=2.0.2
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
WARNING: The candidate selected for download or install is a yanked version: 'python-levenshtein' candidate (version 0.12.0 at https://files.pythonhosted.org/packages/42/a9/d1785c85ebf9b7dfacd08938dd028209c34a0ea3b1bcdb895208bd40a67d/python-Levenshtein-0.12.0.tar.gz#sha256=033a11de5e3d19ea25c9302d11224e1a1898fe5abd23c61c7c360c25195e3eb1 (from https://pypi.org/simple/python-levenshtein/))
Reason for being yanked: Insecure, upgrade to 0.12.1
Building wheels for collected packages: annoy, python-Levenshtein, scispacy, awscli, langdetect, nltk, pyDatalog, python-docx, sentence-transformers, sacremoses
Building wheel for annoy (setup.py) ... done
Created wheel for annoy: filename=annoy-1.16.3-cp37-cp37m-win_amd64.whl size=48250 sha256=6d2d542f78b946e1cdb75f2727bf5474042c8bcf1c72d3b44386ba24bf6b6af6
Stored in directory: pip\cache\wheels\39\36\d4\ee348a7240ca3e8d1fcbf04ebe46d45f2879ccb094a40f5706
Building wheel for python-Levenshtein (setup.py) ... done
Created wheel for python-Levenshtein: filename=python_Levenshtein-0.12.0-cp37-cp37m-win_amd64.whl size=81989 sha256=7ef7a28f7b36c10d775dc493addc63fd7ed936de8db65b2631724853ab6a4ee9
Stored in directory: pip\cache\wheels\f0\9b\13\49c281164c37be18343230d3cd0fca29efb23a493351db0009
Building wheel for scispacy (setup.py) ... done
Created wheel for scispacy: filename=scispacy-0.2.4-py3-none-any.whl size=35204 sha256=08c8526b27c5c4b6f2f8b78bba973fe2a1d617959cf4573a880ba9b510453e35
Stored in directory: pip\cache\wheels\96\24\54\8c2bf5a6804275431a74bd256821500b3f2911d9ba8470a846
Building wheel for awscli (setup.py) ... done
Created wheel for awscli: filename=awscli-1.19.12-py2.py3-none-any.whl size=3564391 sha256=095fbe4e66cdde5eba3d4edd7801a61d9dfce726337cb88d6999ed05cc4801c1
Stored in directory: pip\cache\wheels\ae\6d\8e\fe9fcfdc1d0bb4a01c0ab812eb1f6d191d872314bf3ae71279
Building wheel for langdetect (setup.py) ... done
Created wheel for langdetect: filename=langdetect-1.0.8-py3-none-any.whl size=993190 sha256=8d5ea79fa6c45a021f03f4dab474e1cc35ca186f855dd4af125a99afa28bf92c
Stored in directory: pip\cache\wheels\59\f6\9d\85068904dba861c0b9af74e286265a08da438748ee5ae56067
Building wheel for nltk (setup.py) ... done
Created wheel for nltk: filename=nltk-3.5-py3-none-any.whl size=1434677 sha256=a2d7879ea0e39222e4ec56fa2793c4256da60c774e4c1ddb45f8c1a8356688c8
Stored in directory: pip\cache\wheels\45\6c\46\a1865e7ba706b3817f5d1b2ff7ce8996aabdd0d03d47ba0266
Building wheel for pyDatalog (setup.py) ... done
Created wheel for pyDatalog: filename=pyDatalog-0.17.1-py2.py3-none-any.whl size=74197 sha256=537a7766fbb9a172e10b2363d373f33855bbb9134aac45024246397ea5ea195d
Stored in directory: pip\cache\wheels\04\e7\e1\8fa2c8dfe74d0b9e20818c8ec00c20657baa0738976c93e46e
Building wheel for python-docx (setup.py) ... done
Created wheel for python-docx: filename=python_docx-0.8.10-py3-none-any.whl size=184489 sha256=d1aee6204eb909a271ee69dc52a9ea4423859489aa39ebf7cdc0954cbf8a0481
Stored in directory: pip\cache\wheels\75\c6\69\05491f32dc052cd70476b65f5bf7082a9b274045f6b001b821
Building wheel for sentence-transformers (setup.py) ... done
Created wheel for sentence-transformers: filename=sentence_transformers-0.4.1.2-py3-none-any.whl size=103066 sha256=2eee4fc5b4566a2e54fa330bcf05cb5c39946934f1a83b0a909b77b98379d6a6
Stored in directory: pip\cache\wheels\1c\a2\db\5d9e7c4aa8dbd82718c202fb1bc7118c2d3bf0925af92943f2
Building wheel for sacremoses (setup.py) ... done
Created wheel for sacremoses: filename=sacremoses-0.0.43-py3-none-any.whl size=893258 sha256=1a11654e208885cc7fa2c6f64fd4d0e3a9cc5c338d1b78dc26282e91ab1ac355
Stored in directory: pip\cache\wheels\69\09\d1\bf058f7d6fa0ecba2ce7c66be3b8d012beb4bf61a6e0c101c0
Successfully built annoy python-Levenshtein scispacy awscli langdetect nltk pyDatalog python-docx sentence-transformers sacremoses
Installing collected packages: six, zipp, urllib3, typing-extensions, python-dateutil, numpy, murmurhash, jmespath, cymem, wasabi, tqdm, srsly, regex, pyparsing, pyasn1, preshed, plac, joblib, importlib-metadata, idna, click, chardet, botocore, blis, tokenizers, threadpoolctl, thinc, scipy, sacremoses, s3transfer, rsa, requests, PyYAML, pybind11, psutil, packaging, filelock, docutils, colorama, catalogue, transformers, torch, spacy, sentencepiece, scikit-learn, pysbd, nmslib, nltk, lxml, conllu, awscli, sentence-transformers, scispacy, python-Levenshtein, python-docx, pyDatalog, nest-asyncio, langdetect, fuzzywuzzy, annoy, foo
Running setup.py develop for foo
Successfully installed PyYAML-5.4.1 annoy-1.16.3 awscli-1.19.12 blis-0.4.1 botocore-1.20.12 catalogue-1.0.0 chardet-4.0.0 click-7.1.2 colorama-0.4.3 conllu-4.4 cymem-2.0.5 docutils-0.15.2 filelock-3.0.12 foo fuzzywuzzy-0.15.0 idna-2.10 importlib-metadata-3.4.0 jmespath-0.10.0 joblib-1.0.1 langdetect-1.0.8 lxml-4.6.2 murmurhash-1.0.5 nest-asyncio-1.4.2 nltk-3.5 nmslib-2.1.1 numpy-1.20.1 packaging-20.9 plac-1.1.3 preshed-3.0.5 psutil-5.8.0 pyDatalog-0.17.1 pyasn1-0.4.8 pybind11-2.6.1 pyparsing-2.4.7 pysbd-0.3.4 python-Levenshtein-0.12.0 python-dateutil-2.8.1 python-docx-0.8.10 regex-2020.11.13 requests-2.25.1 rsa-4.5 s3transfer-0.3.4 sacremoses-0.0.43 scikit-learn-0.24.1 scipy-1.6.1 scispacy-0.2.4 sentence-transformers-0.4.1.2 sentencepiece-0.1.85 six-1.15.0 spacy-2.2.3 srsly-1.0.1 thinc-7.3.1 threadpoolctl-2.1.0 tokenizers-0.10.1 torch-1.7.1 tqdm-4.57.0 transformers-4.3.2 typing-extensions-3.7.4.3 urllib3-1.26.3 wasabi-0.8.2 zipp-3.4.0
Actually it's interesting that it installs python-Levenshtein 0.12.0
which has been yanked: https://pypi.org/project/python-Levenshtein/0.12.0/
I guess if the requirements explicitly state that version it gets installed anyway.
Looks like langdetect 1.0.8 has a better dependency tree. Maybe changing it to langdetect<1.0.8
would reproduce. (This is one of the beautiful thing working on this, it's a pain to reproduce anything since there are so many moving parts.)
Looks like langdetect 1.0.8 has a better dependency tree. Maybe changing it to
langdetect<1.0.8
would reproduce. (This is one of the beautiful thing working on this, it's a pain to reproduce anything since there are so many moving parts.)
Indeed, tried adding that but still installed fine. Rereading the initial report it looks like they didn't fully list their requirements as the first item in the log states Collecting Flask==0.12.2
but that's not listed in the requirements they have given. But even using the following:
from setuptools import setup
setup(
name="foo",
version="1.0",
install_requires=[
'Flask==0.12.2',
'spacy==2.2.3',
'sentence-transformers',
'sentencepiece==0.1.85',
'scispacy>=0.2.0',
'nltk',
'python-docx',
'langdetect<1.0.8',
'fuzzywuzzy==0.15.0',
'python-Levenshtein==0.12.0',
'annoy==1.16.3',
'pyDatalog',
'srsly==1.0.1',
'nest_asyncio==1.4.2']
)
It still installed fine, in fact reading the original post further I think there are many more packages that haven't listed, but reverse engineering their requirements from their logs is too laborious for me to try further.
I thought I had a good idea for speeding up these kind of situations, but finding a good test case sure is hard!
I'm encountering this issue on a very small requirements.txt
with only two packages:
spacy==2.3.5
spacy-pkuseg
From inspecting their dependencies manually, I can see very clearly that there is an incompatible dependency on srsly
:
spacy 2.3.5
: srsly>=1.0.2,<1.1.0
spacy-pkguseg
: srsly>=2.3.0,<3.0.0
It looks like pip gets stuck backtracking on a whole bunch of different things. If there is no valid solution and if we don't think pip should install an invalid solution, then it should at least fail faster so I can move on. There are only three versions of spacy-pkguseg
to check yet pip goes into a crazy spiral of downloading dozens of versions of other dependencies.
Collecting spacy==2.3.5
Using cached spacy-2.3.5-cp38-cp38-macosx_10_9_x86_64.whl (10.2 MB)
Collecting spacy-pkuseg
Using cached spacy_pkuseg-0.0.28-cp38-cp38-macosx_10_9_x86_64.whl (2.4 MB)
Collecting cymem<2.1.0,>=2.0.2
Using cached cymem-2.0.5-cp38-cp38-macosx_10_9_x86_64.whl (31 kB)
Collecting numpy>=1.15.0
Using cached numpy-1.20.1-cp38-cp38-macosx_10_9_x86_64.whl (16.0 MB)
Collecting thinc<7.5.0,>=7.4.1
Using cached thinc-7.4.5-cp38-cp38-macosx_10_9_x86_64.whl (982 kB)
Collecting wasabi<1.1.0,>=0.4.0
Using cached wasabi-0.8.2-py3-none-any.whl (23 kB)
Collecting catalogue<1.1.0,>=0.0.7
Using cached catalogue-1.0.0-py2.py3-none-any.whl (7.7 kB)
Requirement already satisfied: setuptools in /Users/jqi/miniconda3/envs/pip-conflict/lib/python3.8/site-packages (from spacy==2.3.5->-r req-test.txt (line 1)) (49.6.0.post20210108)
Collecting murmurhash<1.1.0,>=0.28.0
Using cached murmurhash-1.0.5-cp38-cp38-macosx_10_9_x86_64.whl (18 kB)
Collecting plac<1.2.0,>=0.9.6
Using cached plac-1.1.3-py2.py3-none-any.whl (20 kB)
Collecting tqdm<5.0.0,>=4.38.0
Using cached tqdm-4.57.0-py2.py3-none-any.whl (72 kB)
Collecting preshed<3.1.0,>=3.0.2
Using cached preshed-3.0.5-cp38-cp38-macosx_10_9_x86_64.whl (105 kB)
Collecting requests<3.0.0,>=2.13.0
Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting blis<0.8.0,>=0.4.0
Using cached blis-0.7.4-cp38-cp38-macosx_10_9_x86_64.whl (5.8 MB)
Collecting srsly<1.1.0,>=1.0.2
Using cached srsly-1.0.5-cp38-cp38-macosx_10_9_x86_64.whl (177 kB)
Collecting chardet<5,>=3.0.2
Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: certifi>=2017.4.17 in /Users/jqi/miniconda3/envs/pip-conflict/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy==2.3.5->-r req-test.txt (line 1)) (2020.12.5)
Collecting spacy-pkuseg
Using cached spacy_pkuseg-0.0.27-cp38-cp38-macosx_10_9_x86_64.whl (2.0 MB)
Using cached spacy_pkuseg-0.0.26-cp38-cp38-macosx_10_9_x86_64.whl (2.0 MB)
INFO: pip is looking at multiple versions of wasabi to determine which version is compatible with other requirements. This could take a while.
Collecting wasabi<1.1.0,>=0.4.0
Using cached wasabi-0.8.1-py3-none-any.whl (23 kB)
Using cached wasabi-0.8.0-py3-none-any.whl (23 kB)
Using cached wasabi-0.7.1.tar.gz (22 kB)
Using cached wasabi-0.7.0.tar.gz (22 kB)
Using cached wasabi-0.6.0-py3-none-any.whl (20 kB)
Using cached wasabi-0.5.0-py3-none-any.whl (20 kB)
Using cached wasabi-0.4.2-py3-none-any.whl (20 kB)
Using cached wasabi-0.4.1-py3-none-any.whl (20 kB)
Using cached wasabi-0.4.0-py3-none-any.whl (20 kB)
INFO: pip is looking at multiple versions of wasabi to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
Using cached urllib3-1.26.1-py2.py3-none-any.whl (136 kB)
Using cached urllib3-1.26.0-py2.py3-none-any.whl (136 kB)
Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Using cached urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Using cached urllib3-1.25.7-py2.py3-none-any.whl (125 kB)
Using cached urllib3-1.25.6-py2.py3-none-any.whl (125 kB)
Using cached urllib3-1.25.5-py2.py3-none-any.whl (125 kB)
Using cached urllib3-1.25.4-py2.py3-none-any.whl (125 kB)
Using cached urllib3-1.25.3-py2.py3-none-any.whl (150 kB)
Using cached urllib3-1.25.2-py2.py3-none-any.whl (150 kB)
Using cached urllib3-1.24.3-py2.py3-none-any.whl (118 kB)
Using cached urllib3-1.24.2-py2.py3-none-any.whl (131 kB)
Using cached urllib3-1.24.1-py2.py3-none-any.whl (118 kB)
Using cached urllib3-1.24-py2.py3-none-any.whl (117 kB)
Using cached urllib3-1.23-py2.py3-none-any.whl (133 kB)
Using cached urllib3-1.22-py2.py3-none-any.whl (132 kB)
Using cached urllib3-1.21.1-py2.py3-none-any.whl (131 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
Collecting tqdm<5.0.0,>=4.38.0
Using cached tqdm-4.56.2-py2.py3-none-any.whl (72 kB)
Using cached tqdm-4.56.1-py2.py3-none-any.whl (72 kB)
Using cached tqdm-4.56.0-py2.py3-none-any.whl (72 kB)
Using cached tqdm-4.55.2-py2.py3-none-any.whl (68 kB)
Using cached tqdm-4.55.1-py2.py3-none-any.whl (68 kB)
Using cached tqdm-4.55.0-py2.py3-none-any.whl (68 kB)
Using cached tqdm-4.54.1-py2.py3-none-any.whl (69 kB)
Using cached tqdm-4.54.0-py2.py3-none-any.whl (69 kB)
Using cached tqdm-4.53.0-py2.py3-none-any.whl (70 kB)
Using cached tqdm-4.52.0-py2.py3-none-any.whl (71 kB)
Using cached tqdm-4.51.0-py2.py3-none-any.whl (70 kB)
Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB)
Using cached tqdm-4.50.1-py2.py3-none-any.whl (70 kB)
Using cached tqdm-4.50.0-py2.py3-none-any.whl (70 kB)
Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
Using cached tqdm-4.48.2-py2.py3-none-any.whl (68 kB)
Using cached tqdm-4.48.1-py2.py3-none-any.whl (68 kB)
Using cached tqdm-4.48.0-py2.py3-none-any.whl (67 kB)
Using cached tqdm-4.47.0-py2.py3-none-any.whl (66 kB)
Using cached tqdm-4.46.1-py2.py3-none-any.whl (63 kB)
Using cached tqdm-4.46.0-py2.py3-none-any.whl (63 kB)
Using cached tqdm-4.45.0-py2.py3-none-any.whl (60 kB)
Using cached tqdm-4.44.1-py2.py3-none-any.whl (60 kB)
Using cached tqdm-4.44.0-py2.py3-none-any.whl (60 kB)
Using cached tqdm-4.43.0-py2.py3-none-any.whl (59 kB)
Using cached tqdm-4.42.1-py2.py3-none-any.whl (59 kB)
Using cached tqdm-4.42.0-py2.py3-none-any.whl (59 kB)
Using cached tqdm-4.41.1-py2.py3-none-any.whl (56 kB)
Using cached tqdm-4.41.0-py2.py3-none-any.whl (56 kB)
Using cached tqdm-4.40.2-py2.py3-none-any.whl (55 kB)
Using cached tqdm-4.40.1-py2.py3-none-any.whl (55 kB)
Using cached tqdm-4.40.0-py2.py3-none-any.whl (54 kB)
Using cached tqdm-4.39.0-py2.py3-none-any.whl (53 kB)
Using cached tqdm-4.38.0-py2.py3-none-any.whl (53 kB)
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of thinc to determine which version is compatible with other requirements. This could take a while.
Collecting thinc<7.5.0,>=7.4.1
Downloading thinc-7.4.3-cp38-cp38-macosx_10_9_x86_64.whl (981 kB)
|████████████████████████████████| 981 kB 3.4 MB/s
Downloading thinc-7.4.2-cp38-cp38-macosx_10_9_x86_64.whl (2.1 MB)
|████████████████████████████████| 2.1 MB 3.4 MB/s
Using cached thinc-7.4.1-cp38-cp38-macosx_10_9_x86_64.whl (2.1 MB)
Collecting blis<0.8.0,>=0.4.0
Using cached blis-0.4.1-cp38-cp38-macosx_10_9_x86_64.whl (3.7 MB)
^CERROR: Operation cancelled by user
@uranusjr Maybe I'm being naive but it seems looking at the above example pip should be able to fail / resolve much faster.
Here is the test case:
from setuptools import setup
setup(
name="foo",
version="1.0",
install_requires=[
'spacy==2.3.5',
'spacy-pkuseg'
]
)
Currently as explained above pip gets stuck here. By hand I can resolve this as impossible by doing the following:
In this example it would look like this:
Would this breadth first search through the requirements decency tree not be a much faster resolution in lots of failure cases? Or am I being naive in the scope of the problem or the complexity to implement?
Breadth-first search is definitely an interesting proposal (and IIRC was raised before), but the idea of a “depth” is a bit more complicated past the most simplistic examples because the dependencies usually form a graph instead of a tree. Nowhere technically infeasible (not even that difficult) though, so I’d definitely encourage you to look into the possibility.
The biggest issue to the approach in my mind is not about the implementation, but to somehow know whether it is an improvement. It obviously improves the example here, and likely many others showing a similar failure pattern, but would it worsen cases that run just fine right now? If it does, is the improvement enough to justify the degradation? How are we even supposed to put cases on a scale? Those are the more important questions to answer than the implementation itself.
Yeah, I'm not sure that the breadth first search would be better in all/most cases. I think the main thing is collecting all packages where there can only be 1 possible version installed and then trying not to be "dumb" in the search methodology through the remaining packages (breadth first was my first idea but I have other ideas).
Understood on the problems you've outlined, it's hard to know if you're making a net positive or a net negative or if you've made certain edge cases 1000x worse. But I have some ideas on testing this as well.
I'll start taking a look at the resolver.resolve
and see if I can figure out how to tweak the implementation to test ideas.
Last I'd tried breadth first search, I'd found the results to be a bit surprising and mote difficult to reason about than depth first search. I also didn't find any package manager that approached the problem that way. That was 2 years ago tho.
FYI I'm no longer able to reproduce the issue with:
spacy==2.3.5
spacy-pkuseg
I've got a new optimization I am testing, so if anyone has a reproducible example let me know.
@notatallshaw which version of pip are you using? I am encountering this issue :/
@daniellepintz They're working on making improvements to pip's behaviour in this area, and I believe that was a comment about the improved behaviour they're seeing on the local in-development copy of pip that they have.
@notatallshaw which version of pip are you using? I am encountering this issue :/
Do you have a reproducible set of requirements?
As @pradyunsg mentions I am working on what should be a performance improvement when people encounter this issue, but I am always looking for more reproducible examples to confirm it actually works.
Someone just posted an example here of azure-cli with Python 3.9 on Linux but then deleted the comment. I tried in a fresh virtual environment but wasn't albe to reproduce.
In general I think this issue should be closed, all reported and reproducible examples were fixed by https://github.com/pypa/pip/issues/10479
New examples should be new issues, and feel free to CC me and I'll check if I can reproduce and see what the issue is.
What did you want to do? We run the command:
on a CI node with no previous cache using a virtual environment.
There was NO cache and there were NO previous packages. We are using a new virtual environment
Python 3.7.3
setup.py
Output
Additional information