Open cdeil opened 7 years ago
@piotr-dobrogost - Yes, I managed to do what I wanted using this:
pip install . --user --no-deps -I
So there is certainly a workaround.
The question from my side remains -- is this intended / good behaviour by pip?
It's intended, because Python (not pip, but Python's core import mechanisms) doesn't really like multiple versions of a package on sys.path
. Consider if foo
version 1.0 included foo/bar.py
but foo
version 2.0 removed it. You could then still import foo, foo.bar
, but would get the v2.0 version of foo
and the v1.0 version of bar
.
Basically, pip uninstalls older versions precisely to avoid this issue. The --ignore
flag says "I know what I'm doing, and take responsibility for any issues caused by having both versions on sys.path
together".
This is a recent change in behaviour from pip though, no?
(to try to uninstall stuff in system site-packages when I say pip install something --user
)
Was it very recently? Is it visible in the pip release notes? Am I the first one to be puzzled by this?
Personally I'm fine with the old or new behaviour.
So just close this issue? Or is there something that can / should be improved about the pip release notes or docs? Is there a way to "follow" such updates / changes in pip, like an RSS feed or blog?
TBH, I'm not sure about that. I never put packages in both places, so I can't honestly say from my own experience, and I'd have to trawl the changelogs to see if there was any specific mention.
I'm not sure if you're the first one to be caught by this - I know I've explained the "you shouldn't have multiple versions on sys.path" issue before, but I can't recall if it was in relation to this or some other scenario.
If you propose a change to pip, leave the issue open and your suggestion can be looked at - but my personal view is that the current behaviour is fine (which I think you're OK with). If you think a doc change would help, we'd be very grateful for a PR with your suggested rewording. If we missed including something in the changelogs, then it's probably water under the bridge now (I don't know if we have a policy on retroactively updating the changelog). Going forward, the changelog is definitely where you'd keep up to date with what's changed as new versions come out.
I don't know if it's a good idea to start uninstalling stuff when a user says install. My guess is that there will be surprises and gotchas either way.
But if that's the intended behaviour, fine. I'm still surprised I haven't encountered this before and didn't have to use --ingore-installed
for the past few years where I've been using pip.
I had a look at the docs and found it hard to find a description of the behaviour of pip. Here it says
Install the packages (and uninstall anything being upgraded/replaced). https://pip.pypa.io/en/stable/reference/pip_install/?highlight=ignore-installed#overview
Searching for ignore-installed
I found this:
https://pip.pypa.io/en/stable/user_guide/?highlight=ignore-installed#user-installs
pip install --user follows four rules:
- When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.
- When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied (similar to how global packages can satisfy requirements when installing packages in a --system-site-packages virtualenv).
It's not obvious to me from that description that pip will basically always try to uninstall the given package I'm trying to install, if it's anywhere on my sys.path
. Maybe that section could be the place to point that behaviour out more explicitly?
(there already is an example using --ignore-installed
at the very bottom of the section)
Or maybe there could be another mention of this behaviour and of --ignore-installed
already earlier in the pip docs? Mentioning that pip install will try to uninstall stuff and will commonly fail if one doesn't have write premissions for system packages could even go in the quickstart IMO.
@pfmoore and @piotr-dobrogost - Close issue now? Or do you agree that some docs edits (which ones exactly) could be made to improve the description?
@piotr-dobrogost seems to have found some of the history (although for some reason I can only see his comment in email, not here). Looks like it is changed behaviour, and was requested by some people. Agreed it should be documented better (but I don't understand the original details of the change, so can't really say how to describe the current situation). Sorry, I can't really offer more help.
I agree behavior of install --user
is not intuitive to say the least. Well, to say the truth I think it's plainly wrong. I remember making comments to this effect in the past here. As the whole user scheme itself is defined so that its location comes before the location of site-packages on sys.path there should be no check performed of what's already installed in site-packages when using user scheme for installation. The only thing worth considering is what to do when the package had already been installed using user scheme. Although it's not clear for me if it's even technically possible to establish what's installed in user scheme vs what's installed in site-packages; otherwise the problem such as How to uninstall a package installed with pip install --user wouldn't have existed I guess. However I guess that after recent change in setuptools and finally getting rid of sys.path mangling performed in easy_install.pth it should be safe just to install in user scheme no matter if it's already installed there.
@pfmoore You should immediately delete this email as I deleted my comment which you received in this email :) I deleted the comment as it was talking about editable install not user scheme one.
Ah, that's what happened! Sorry. Github emails don't reflect edits, I've been bitten by that before...
Agreed what happens is unintuitive, and I wouldn't be opposed if someone argued it's wrong. However, as I noted, having 2 different versions in system site-packages and user site-packages could cause issues. We can't win either way - someone's going to be bitten by unintuitive behaviour no matter what we do.
Ideally, someone should review this whole area and propose a resolution - which may be documentation only or may include behaviour changes. But without any funded support for pip, that's not likely to happen unless a user who's actually experiencing issues steps up to do the work.
(...) having 2 different versions in system site-packages and user site-packages could cause issues.
It should not as the whole point of user site-packages is to take precedence over global site-packages; the two are meant to be distinct and separate namespaces. The only problem is pip's wrong implementation of install --user
.
It should not as the whole point of user site-packages is to take precedence over global site-packages
Well, yes - but that's not how Python works. We can happily reject bug reports as "not a pip issue", but that still leaves us with confused users.
If you want to raise a bug against Python, I guess you could. I doubt it'd be easy to get it fixed, though.
The behaviour I had for the past years and got used to was that pip install
never uninstalls, but just puts new versions. And then somehow Python import always picks up the newest. It was OK for me. The new, current behaviour is also reasonable, other package managers do the same thing to remove older versions before putting the new one.
I probably won't participate further in discussions and don't have time to contribute code or docs fixes to pip, sorry.
So just to summarise, from the user perspective, my main wish here is to get more information about how pip works and when new releases come out, to explain better what behaviour changed (via a "What's new page" or blog post). One presentation I found very interesting was http://www.dabeaz.com/modulepackage/ which explains about Python modules and packages and imports. If someone here has time to put together a similar "deep dive" detailed talk or tutorial for pip in the future, I would find that super interesting.
Well, yes - but that's not how Python works.
I see, you are referring here to the contrived example with foo.bar
you gave earlier. Well, I didn't think of this corner case earlier. This indeed justifies considering globally installed package when installing the package in user scheme. Albeit I would say it's a bug in Python to take foo from one place and foo.bar from another not related to the first. Upon importing foo the import logic should lock import location for all subsequent imports beginning with foo. Now, when I'm aware of this unfortunate behavior of import mechanics in Python I'm wondering what were the authors of user scheme thinking when they created the scheme?
They probably assumed people wouldn't install in both system packages and user packages. Or maybe they just didn't think of this issue. Who knows? And yes, my example was contrived. However, I think (although I can't be sure, or provide references) that we've had real issues as a result of this.
On a purely personal note, I only use the system site-packages (I'm on Windows, and I have a per-user install of Python 3.6, so this doesn't affect anything else like a system package manager) and virtualenvs, so the issue doesn't affect me either way (apart from needing to provide support to people on issues like this). So I'll leave this discussion now to be taken further by people who are affected.
@dstufft How is user scheme meant to work in light of problem described in https://github.com/pypa/pip/issues/4337#issuecomment-286786322 ?
@piotr-dobrogost It works fine if you install a given package in only user or system location, not in both. Sorry I may have misunderstood as I assumed that you realised that. It's only when you try to install different versions the same distribution in both places that there's any possibility of an issue.
@piotr-dobrogost My apologies. Over on distutils-sig, @ncoghlan corrected me on this. Apparently the import system does protect against multiple version clashes like this. At least to the extent that the first version that's on sys.path will always "win", so there's never going to be inconsistent imports like I described.
I couldn't find anything in the docs about whether the user scheme comes before or after the system scheme in sys.path. I assume it comes before, so that user installs take precedence over system installs, but I've been bitten once today by not testing my claims, so I'll just say "I assume" here... :-(
Yea, it goes:
.
).In each of those, I think any .pth
files get added immediately after the directory that introduced the .pth
file.
Over on distutils-sig, @ncoghlan corrected me on this.
Can you share link to this?
Apparently the import system does protect against multiple version clashes like this.
Ok. In that case we are back to pip's behavior and my earlier statement that pip shouldn't care what's already installed globally when installing in user scheme. @dstufft Would you agree?
- When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.
means pip install --user ...
never uninstalls. Globally installed packages (in system site-packages
) should be ignored, not uninstalled, and the new package should be installed in the user scheme which will be found and imported first.
So this appears to be a regression in 9.x and in contradiction with the docs.
As far as I understand it, the whole point of the user scheme is that users may not have control over system packages, and may not be able to uninstall anything from there.
Even if they did have permission, one user installing a new version of a package in their user namespace, triggering an uninstall from system site-packages
, would break that package for all other users on that system.
Globally installed packages (in system site-packages) should be ignored (...)
They should but unless you pass --ignore-installed
that's sadly not the case which is very confusing. In practice pip does not see difference between packages installed in user namespace and in system site-packages.
@piotr-dobrogost but if I use --ignore-installed
, all my requirements will be (re)installed into the userbase directory. Not only the ones that aren't already satisfied by what is already installed in system site-packages
. Again, this is in contradiction with the docs, and is different to previous behaviour of pip?
- When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied (similar to how global packages can satisfy requirements when installing packages in a --system-site-packages virtualenv).
When the first two of four rules for the way a feature operates are completely wrong, and the remaining rules are basically just sanity checks preventing action where it wouldn't make sense anyway, it seems there's not much point in supporting the user scheme at all.
I'm a bit confused though, about why I'm pretty sure this was working as expected and as documented before, but is now completely broken.
Related #4575
@cdeil would it be an improvement to have pip send a message on stderr
that tells the user that it is deleting the older version? That seems like a simple fix that would make it clear to the user what is happening. That message would also give users that don't want this behavior a string to search on Google that would help them find the -I
flag.
The --ignore-installed flag isn't a exact fix for the problem here.
Ha ha, I haven't used --user
since 2 years, switched to use conda envs and conda and pip install things there and that works fine. So personally I don't really care about --user
any more.
But re-reading the discussion above: yeah, I would still say a PermissionError
from system site packages when trying to pip install --user
something is a bug. Similar to what others said above, pip install --user
should just install the package and not look at all what's in system site-packages. I think the initial counter-argument that foo.bar
gets mixes is not true, Python doesn't do that, it's OK to have the same package installed multiple times in different envs in the sys.path
list, like in this case system site-packages and now user site-packages.
@cdeil would it be an improvement to have pip send a message on stderr that tells the user that it is deleting the older version?
Well, yes, if that is an easy addition, and the real fix and change in behaviour of pip install --user
to not delete isn't coming within the near future (say ~ 1 year), then you could add that message now.
Description:
I've been a happy user of pip for a while, thank you for all your work!
Now (possibly after a recent pip update), the bahaviour has changed and
from a source folder in a Python project I work on (Gammapy) tries to uninstall a stable 0.5 version of Gammapy I have in the Macports site-packages.
I wasn't expecting pip to do this uninstall and it's not clear to me from the pip 9.0 and 9.0.1 release notes that this is an intended change in behaviour.
Is this a bug or am I doing it wrong? What's the workaround?
What I've run: