Pip cannot resolve recursive dependencies

[dizcza]

I'm trying to install pip install .[extras,tutorials] from this branch: https://github.com/INM-6/elephant/blob/docs/more/setup.py

Elephant requirements/requirements-tutorials.txt contains a line viziphant @ git+https://github.com/INM-6/viziphant.git but viziphant, in turns, requires elephant[extras] package. Such a recursive dependency breaks the pip: see the full log here; you can reproduce it locally by running

pip install git+https://github.com/INM-6/elephant@docs/more#egg=elephant[extras,tutorials]

Pip 20.2.4 doesn't have such an issue.

Pip 20.3 and the master branch have this issue.

Originally asked in https://github.com/readthedocs/readthedocs.org/issues/7721.

I took a look at https://github.com/pypa/pip/issues/8713 - at least it stops at some point, while this issue makes an infinite loop.

[brainwane]

Hello and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us.

While we're waiting for the resolver developers to look at your report and respond, I'll mention here one useful troubleshooting and workaround tip from the documentation:

If you run into resolution errors and need a workaround while you’re fixing their root causes, you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

(If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the pip resolver beta period.)

[dizcza]

I'm aware of the legacy flag, though it might be difficult to incorporate this on readthedocs; I give it a try.

If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the pip resolver beta period.

Sorry, I don't get what you mean. I didn't try v20.3 beta and I was unaware of these changes before I encountered this bug.

[uranusjr]

I think what Sumana was trying to ask were

• Did you know about the change before 20.3 was released?
• Would you test your usages (and thus encountered the issue before 20.3) if you know about the change?
• Would you expect anything different from pip, if the issue was reported before 20.3?

[joestringer]

I was able to work around this issue locally by using the flag --use-deprecated=legacy-resolver, thanks for the suggestion @brainwane . Unfortunately like @dizcza , we also build our docs on readthedocs.org and I can't see any way for us to influence those builds in the same way so they will remain broken until this is resolved. On the plus side, assuming that this is resolved and a new version of pip is released, it looks like readthedocs will automatically pull the new version so in theory the problem there should "resolve itself" without additional readthedocs user interaction.

One other point that may be useful, here's the line in the corresponding requirements.txt file that triggers this issue:

git+git://github.com/cilium/sphinx_rtd_theme.git@v0.7; platform_machine != "aarch64"

[uranusjr]

What is the error you’re seeing? I can’t view the RTD build log (probably a permission issue), and the behaviour I get from running the pip install command locally does not match your characterisation to the issue.

[dizcza]

@uranusjr pip.log (I aborted the infinite loop after a few seconds of the run)

• Did you know about the change before 20.3 was released? - Nope.
• Would you test your usages (and thus encountered the issue before 20.3) if you know about the change? - I'd test it, of course, though the problem appeared only with the latest changes where I added the offending line in requirements-tutorials.txt. The master branch of elephant works with 20.3 because there is no recursion there ... yet.
• Would you expect anything different from pip, if the issue was reported before 20.3? - I hope that you guys know what you're doing. If you decide to change the policy, so be it. I'm not into pip, so I don't know what one could expect from pip if I reported it a while ago.

[uranusjr]

Thanks! The output seems to match my local result; the resolver can solve the dependencies, but it requires a lot of time (and resources) because there are incompatible version requirements in the set that needs expensive combining. I believe this is ultimately #9187, so I’ll merge the conversation there.

[brainwane]

I didn't try v20.3 beta

@dizcza thanks for this reply. To help us roll out changes in the future: did you know about the 20.3b1 beta, and/or about the beta version of the resolver available in pip 20.2? (I'm trying to figure out: did you know about the beta and decide not to test it, or did you just not hear about it?) And if you didn't hear about it at all, can you tell us where you get your news about things that will affect you as a developer, so we can try to publicize stuff there in the future? We have a low-traffic announce-only mailing list you might want to subscribe to, for major pip & PyPI changes.

[dizcza]

@brainwane well, I treat pip as a bible: if it breaks, what can I do? It cannot ...fail

I didn't know about the upcoming changes; perhaps, I intentionally avoided looking at warning messages if there were any from pip 20.2.x. Pip, as many other build/packae tool projects that I'm using (make, cmake, pocl, gcc, cuda tools), is out of scope of my interests: I cannot possible track all of them, even if I wanted. Such a change that pip is currently undergoing is a rare case. I'll definitely test the fix that you come up with, if any, and probably the next beta releases. Though I refraim myself for subscribing to such channels as pip mailing list unless I decide to subscribe it for the current months, which sounds a reasonable thing to do.

[pfmoore]

@dizcza As a related question, how do you manage the process of updating your build tools like pip? Obviously you pick up the new version, as that's what triggered the report, but do you test the new version locally before using it in your main workflow, or do you just grab the latest version? Does the way you manage pip versions differ from the other tools you mention like gcc, or do you use the latest version of them when they get released in the same way?

If you do handle pip differently than other tools, is there anything about how pip does its releases that makes it harder for you to stick with a "known good" version until you're ready to upgrade?

[dizcza]

@pfmoore I just grab the latest versions ^-^ That's what I do as a user of the abovementioned build tools. If I find an issue there, which happens rarely, I just report the issue and, in the meanwhile, I'm using the version I find most convenient - either a manual fix or an old build - 'cause I don't need to care about others. Regarding pip, I'm a developer of a tool that other users download, hence I need to stick to common sense. I wouldn't bother with upgrading the pip to the newest version if readthedocs didn't require the latest pip by default to build a package! In fact, that's where I got familiar with the pip updates. Of course, I could fix a pip version to an older build in my requirements (because others mentioned in this discussion that it's currently not possible to pass a flag in readthedocs pip install commands), which I'll probably do a temporary fix, but it's surely not a solution in a long run.

[pfmoore]

Thanks. I think a couple of people have mentioned readthedocs. Maybe it would help readthedocs users to manage their workflow if RTD offered a way to control the version of pip that's used. I don't know if that is something that can be done, or if there's a useful discussion for someone to have with them. But it might be worth affected users exploring that as a possibility.

We (the pip devs) can also think about whether there's any way to better involve RTD in future beta releases.

[dizcza]

Besides, I'm not sure if you guys fix the issue with long dependency resolution runtime in the upcoming version(s). Or fix it partially such that the resolution time drops from half an hour to 3 minutes, which is still long. If I fix the pip version to 20.2.4, then I'm at risk of my package not working at all or installing slowly with the future versions of pip. The only solution is to use the latest pip, which RTD guys are doing by default. If there is a problem, the authors should be notified.

[stsewd]

Hi, at RTD we actually sampled around 1/3 of projects for ~2 months to use the new resolver by default to help users catch any problems (a few reported some problems, but they were already reported in pip).

We install the latest version of pip by default, but users can use a specific version if they want, for that a config file like this is necessary.

version: 2

python:
  version: 3.7
  install:
    # requirements with pip pinned to a specific version (must be first so the other installations use this version of pip)
    - requirements: docs/pip.txt
    # Actual project requirements
    - requirements: docs/requirements.txt

If there is anything else we can do, please let us know