Closed 1kastner closed 10 months ago
To be honest, I am a bit stuck on what I can do here. I would like to see where the conflict stems from - where in the dependency graph do we have the problem?
PS: As this worked in the past, I have a locally working version of it, i.e.:
alabaster==0.7.12
anyio==3.6.1
apeye==1.2.0
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
astroid==2.11.7
asttokens==2.0.8
atomicwrites==1.4.1
attrs==22.1.0
autodocsumm==0.2.9
autopep8==1.6.0
Babel==2.10.3
backcall==0.2.0
beautifulsoup4==4.11.1
black==22.6.0
bleach==5.0.1
Brotli==1.0.9
CacheControl==0.12.11
certifi==2022.6.15
cffi==1.15.1
charset-normalizer==2.1.0
click==8.1.3
colorama==0.4.5
commonmark==0.9.1
-e git+ssh://git@github.com/1kastner/conflowgen.git@cdb6ba2a44c16deaf75b5b424f2e0ab3d554f687#egg=conflowgen
contourpy==1.2.0
coverage==6.4.3
cssselect2==0.6.0
cssutils==2.5.1
cycler==0.11.0
debugpy==1.6.2
decorator==5.1.1
defusedxml==0.7.1
dict2css==0.3.0
dill==0.3.5.1
docutils==0.18.1
domdf-python-tools==3.3.0
entrypoints==0.4
enum-tools==0.9.0.post1
et-xmlfile==1.1.0
execnet==1.9.0
executing==0.10.0
fastjsonschema==2.16.1
flake8==4.0.1
flake8-nb==0.4.0
fonttools==4.34.4
html5lib==1.1
idna==3.3
imagesize==1.4.1
importlib-metadata==4.12.0
iniconfig==1.1.1
ipykernel==6.15.1
ipython==8.4.0
ipython-genutils==0.2.0
isort==5.10.1
jedi==0.18.1
Jinja2==3.1.2
json5==0.9.9
jsonschema==4.9.1
jupyter-client==7.3.4
jupyter-core==4.11.1
jupyter-lsp==1.5.1
jupyter-server==1.18.1
jupyterlab==3.4.5
jupyterlab-code-formatter==1.5.3
jupyterlab-lsp==3.10.1
jupyterlab-pygments==0.2.2
jupyterlab-server==2.15.0
jupyterlab-spellchecker==0.7.2
kaleido==0.2.1
keyring==23.8.2
kiwisolver==1.4.4
latexcodec==2.0.1
lazy-object-proxy==1.7.1
libsass==0.21.0
lockfile==0.12.2
lxml==4.9.1
markdown-it-py==2.1.0
MarkupSafe==2.1.1
matplotlib==3.8.2
matplotlib-inline==0.1.3
mccabe==0.6.1
mdit-py-plugins==0.3.0
mdurl==0.1.2
mistune==0.8.4
msgpack==1.0.4
mypy==0.971
mypy-extensions==0.4.3
myst-parser==0.18.0
natsort==8.1.0
nbclassic==0.4.3
nbclient==0.6.6
nbconvert==6.5.3
nbformat==5.4.0
nbsphinx==0.8.9
nest-asyncio==1.5.5
nose==1.3.7
notebook==6.4.12
notebook-shim==0.1.0
numpy==1.23.2
openpyxl==3.0.10
packaging==21.3
pandas==1.4.3
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.9.0
peewee==3.15.1
pickleshare==0.7.5
Pillow==9.2.0
pkginfo==1.8.3
platformdirs==2.5.2
plotly==5.10.0
pluggy==1.0.0
prometheus-client==0.14.1
prompt-toolkit==3.0.30
psutil==5.9.1
pure-eval==0.2.2
py==1.11.0
pybtex==0.24.0
pybtex-docutils==1.0.2
pycodestyle==2.8.0
pycparser==2.21
pydocstyle==6.1.1
pydyf==0.2.0
pyflakes==2.4.0
Pygments==2.12.0
pylint==2.14.5
pyls-flake8==0.4.0
pyparsing==3.0.9
pyphen==0.12.0
PyQt5==5.15.7
PyQt5-Qt5==5.15.2
PyQt5-sip==12.11.0
pyrsistent==0.18.1
pytest==7.1.2
pytest-cov==3.0.0
pytest-forked==1.4.0
pytest-github-actions-annotate-failures==0.1.7
pytest-xdist==2.5.0
python-dateutil==2.8.2
python-lsp-jsonrpc==1.0.0
python-lsp-server==1.5.0
pytoolconfig==1.2.2
pytz==2022.2.1
pywin32==304
pywin32-ctypes==0.2.0
pywinpty==2.0.7
PyYAML==6.0
pyzmq==23.2.1
readme-renderer==36.0
requests==2.28.1
requests-toolbelt==0.9.1
rfc3986==2.0.0
rich==12.5.1
rope==1.3.0
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
scipy==1.9.0
seaborn==0.11.2
Send2Trash==1.8.0
six==1.16.0
sniffio==1.2.0
snowballstemmer==2.2.0
soupsieve==2.3.2.post1
Sphinx==5.1.1
sphinx-autodoc-typehints==1.19.2
sphinx-jinja2-compat==0.1.2
sphinx-prompt==1.5.0
sphinx-rtd-theme==0.5.1
sphinx-simplepdf==1.3.0
sphinx-tabs==3.4.1
sphinx-toolbox==3.2.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-bibtex==2.4.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
stack-data==0.4.0
tabulate==0.8.10
tenacity==8.0.1
terminado==0.15.0
tinycss2==1.1.1
toml==0.10.2
tomli==2.0.1
tomlkit==0.11.4
tornado==6.2
traitlets==5.3.0
twine==4.0.2
typing_extensions==4.3.0
ujson==5.4.0
urllib3==1.26.11
wcwidth==0.2.5
weasyprint==56.1
webencodings==0.5.1
websocket-client==1.3.3
wrapt==1.14.1
yapf==0.32.0
zipp==3.8.1
zopfli==0.2.1
As I have locall played around with a library or two, this might include some additional packages that are actually not needed for the project.
👋🏽
Try running pip with --log out.log
and look at the output of that to see what it is backtracking on?
Consolidating into https://github.com/pypa/pip/issues/11480, since that's the tracking issue for improving this situation in general. This is a known gap of the tool; and the maximum verbosity logs contain the relevant information (it mentions when backtracking why a conflict/backtracking is happening).
FYI I'm taking at a look at why this set of requirements causes Pip to reach ResolutionTooDeep
I was able to reproduce using steps others will find a bit simpler, though took a couple of hours for me to run on my quite fast computer. Run pip install --dry-run -r requirements.txt
where requirements.txt
is:
scipy
numpy
pandas >=1
openpyxl
peewee >=3
enum_tools >=0.7
matplotlib
plotly
kaleido
pytest
pytest-cov
pytest-xdist
pytest-github-actions-annotate-failures
seaborn
nbformat
nbconvert
sphinx
sphinx-rtd-theme
sphinx-toolbox
myst-parser
sphinxcontrib-bibtex
nbsphinx
ipython
ipykernel
jupyterlab
pylint
flake8 <6.0
flake8_nb
wheel
twine
I would strongly suggest OP add more lower bounds to their requirements. At a quick look at one of the backtracking steps Pip gets stuck on resolving sphinx
and docutils
, with many parents depending on them:
docutils docutils<0.21,>=0.18.1, Parent: sphinx 7.2.6
docutils docutils<0.21, Parent: sphinx-rtd-theme 2.0.0
docutils docutils>=0.16, Parent: sphinx-toolbox 3.5.0
docutils docutils<0.21,>=0.16, Parent: myst-parser 2.0.0
docutils docutils!=0.18.*,!=0.19.*,>=0.8, Parent: sphinxcontrib-bibtex 2.6.1
docutils docutils, Parent: nbsphinx 0.9.3
docutils docutils>=0.14, Parent: pybtex-docutils 1.0.3
docutils docutils>=0.13.1, Parent: readme-renderer 42.0
docutils docutils, Parent: sphinx-prompt 1.8.0
docutils docutils~=0.18.0, Parent: sphinx-tabs 3.4.4
docutils docutils~=0.18.0, Parent: sphinx-tabs 3.4.1
sphinx sphinx
sphinx sphinx<8,>=5, Parent: sphinx-rtd-theme 2.0.0
sphinx sphinx>=3.2.0, Parent: sphinx-toolbox 3.5.0
sphinx sphinx<8,>=6, Parent: myst-parser 2.0.0
sphinx Sphinx>=3.5, Parent: sphinxcontrib-bibtex 2.6.1
sphinx sphinx>=1.8, Parent: nbsphinx 0.9.3
sphinx Sphinx<8.0,>=2.2, Parent: autodocsumm 0.2.11
sphinx sphinx>=7.1.2, Parent: sphinx-autodoc-typehints 1.25.2
sphinx Sphinx<8.0.0,>=7.0.0, Parent: sphinx-prompt 1.8.0
sphinx sphinx<6,>=2, Parent: sphinx-tabs 3.4.0
sphinx sphinx<5,>=2, Parent: sphinx-tabs 3.3.1
sphinx sphinx<5,>=2, Parent: sphinx-tabs 3.3.0
sphinx sphinx<5,>=2, Parent: sphinx-tabs 3.2.0
sphinx sphinx<5,>=2, Parent: sphinx-tabs 3.1.0
sphinx sphinx<5,>=2, Parent: sphinx-tabs 3.0.0
sphinx sphinx<4,>=2, Parent: sphinx-tabs 2.1.0
sphinx sphinx<4,>=2, Parent: sphinx-tabs 2.0.1
sphinx sphinx<4,>=2, Parent: sphinx-tabs 2.0.0
sphinx sphinx<4,>=2, Parent: sphinx-tabs 1.3.0
sphinx sphinx<4,>=2, Parent: sphinx-tabs 1.2.1
The problem here is pip needs to choose a cause or a parent to backtrack on and there are a lot of choices and if the wrong choice is made Pip can spend a lot of time wasted trying to find the right resolution path.
In this example I beleive if pip was intelligent about this step it could realize that sphinx-tabs
requirements are not compatible with the parent of the requirement docutils docutils<0.21,>=0.18.1
sphinx 7.2.6
, so it would choose to backtrack on these, and Pip should then resolve quicker.
I am working on an update to Pip's resolution that prefers backtracking on this conflicting causes: https://github.com/pypa/pip/issues/12318#issuecomment-1835490998. However, I had only considered when two specifiers conflict with each other, not when a parent version and specifier conflict. So I have started working on an update to the branch to handle this scenario and will provide feedback as soon as I have it implemented.
I see in your case you've been able to come up with some constraints that solve your situation, but in general if you have some requirements that used to work but no longer do, one way to identify working constraints is to use pypi-timemachine
as I describe here: https://github.com/pypa/pip/issues/12305#issuecomment-1742218919
Okay, I have updated my experimental Pip branch to prefer conflicting causes to handle this scenario: https://github.com/notatallshaw/pip/tree/prefer-conflicting-causes
If you would like to assist in testing your use case you can install it like so:
pip install git+https://github.com/notatallshaw/pip.git@prefer-conflicting-causes
In my testing this now resolves very quickly.
FYI, for OP you should be able to drastically speed up your resolution in the current version of Pip by including sphinx-tabs
as your last requirement in your dev
extras. This will force the Pip resolver to resolve this requirement and set it on the correct path to resolve all the requirements.
Also for others this simplified set of requirements also demonstrates this pathological behavior in Pip's current resolver:
sphinx
sphinx-rtd-theme
sphinx-toolbox
myst-parser
sphinxcontrib-bibtex
nbsphinx
@notatallshaw thank you for the extensive work! I have tried out your latest solution in https://github.com/1kastner/conflowgen/actions/runs/7159890013/job/19493640829?pr=201 and it worked like a charm. When is it planned to be published?
thank you for the extensive work! I have tried out your latest solution in https://github.com/1kastner/conflowgen/actions/runs/7159890013/job/19493640829?pr=201 and it worked like a charm
Great! It's good to have more examples to push this forward.
When is it planned to be published?
Likely not for awhile, the are a number of steps that need to be done:
In particular I see step 4 as being a bit sticky as I still need to clean up the code, figure out some tests, write up a good explanation for Pip maintainers, formalize all the evidence I have that it is a general improvement, and convince it's worth spending the time to review and merge. It would be great if I did that by the next Pip release, but I could also see it taking a lot longer.
In the mean time I suggest you follow my other workaround:
you should be able to drastically speed up your resolution in the current version of Pip by including sphinx-tabs as your last requirement in your dev extras
Thank you for your guidance! The work-around is in place and l look forward to seeing this improvement to be merged. Best luck for the process!
Description
A set of Python modules that could be installed jointly in the past runs now into
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000
.Expected behavior
A quick installation of all required modules
pip version
23.3.1
Python version
3.8.18
OS
Ubuntu
How to Reproduce
Output
See https://github.com/1kastner/conflowgen/actions/runs/7158163375/job/19489894194?pr=201
Code of Conduct