pradyunsg commented 6 years ago

It seems pip's current upgrade message instructions on Linux doesn't really match what the OS package managers would want users to do -- running pip install --upgrade pip on a Debian/Fedora system can break it (especially when they might end up doing sudo pip or something like that).

It would be nice if we could ask the upstream vendors to modify their version of pip to either not show the message or (better) show the message when the OS vendors the newer version, with appropriate instructions.

For easier future reference; as on 16th July 2018, the following are the people we'd want to ping based on Distro:

Debian: @doko42 @warsaw (neither are "maintainers")
Fedora: @torsava @stratakis @hroncok
Arch Linux: @felixonmars (maintainer), @eli-schwartz (community contributor)

Additionally, we have Conda folks too: @kalefranz @jakirkham @msarahan

pradyunsg commented 6 years ago

I guess that @ncoghlan would be a good person to ask for inputs here.

Doing some research, @warsaw for Debian and @stratakis for Fedora might be the right people to ping here about this.

benoit-pierre commented 6 years ago

You could suppress the message when pip is unvendored, as that's a pretty good way to tell pip is installed from a distribution package manager.

pfmoore commented 6 years ago

While it's not a reason for not doing this, the problem is that we can put such a change in the next version of pip, but the problem is precisely for the cases where the OS version of pip is old, so the fix won't help for a long time.

So while I agree with the change, I think explaining to distro vendors the need for them to customise the selfcheck would be beneficial as well.

ncoghlan commented 6 years ago

At a pip implementation level, I'd suggest checking INSTALLER in pip's own dist-info directory to decide whether or not to run the version check, as that would also add an incentive for distro packagers to ensure that they modify the INSTALLER file appropriately (rather than leaving it as pip, which is what happens when the results of pip-based installation into the build root are copied without modification into the resulting package).

At a documentation level, I also agree with Paul that it would be good to recommend that redistributors patch that message appropriately for their particular system.

stratakis commented 6 years ago

Thanks for creating the issue and notifying us! We recently got bitten by this in Fedora and I guess it was only reported now due to the incompatibilities between pip 9 and pip 10. I'm pretty sure many more people have actually updated pip using that method. We are tracking the issue now downstream and we'll change the message accordingly.

I am also thinking if it would be possible to entirely disable this behaviour (using pip to upgrade pip) while not inside a venv.

hroncok commented 6 years ago

I wasn't aware that this is tracked here, thanks @stratakis for pointing me here. Several things I consider important:

The massage explicitly mentions pip install --upgrade pip -> that would attempt to upgrade the Python 2 pip on Fedora*. The message is like this even when pip3 or python3 -m pip was used to run pip.
The pip install --upgrade pip command does not work on Fedora* - it will fail with Permission error. Users then immediately try with sudo and that can be nasty.
Even python3 -m pip install --user --upgrade pip isn't particularly nice: executing just pip3 after the upgrade still uses /usr/bin/pip3 and that's comes from the old version. (That's probably something we can solve with PATH order.)

However, the message is completely fine in venv. If we are changing it, let's make sure we are not changing it there. I wonder if INSTALLER in pip's own dist-info differs in venv. If it does, @ncoghlan's solution is perfect.

* outside of venv

hroncok commented 6 years ago

I'm working on a patch for Fedora. Will test it and provide a PR.

hroncok commented 6 years ago

Fedora PR first: https://src.fedoraproject.org/rpms/python-pip/pull-request/5

Note that I went with the easy option: I just disabled the warning when INSTALLER doesn't say pip. Allowing different messages based on different INSTALLERs would also require different checking for the newer version (when a new version of pip is released, it takes a while before it is packaged for distros). I felt like the entire thing would need to be redesigned if we would like to allow that. So my patch only went with the simple option: not installed by pip? no warning.

ncoghlan commented 6 years ago

Silencing the warning for non-pip installs in general seems reasonable to me, since redistributors also tend to have their own ways of encouraging upgrades-by-default (e.g. Fedora Workstation points out available updates on start-up, and then has release-monitoring.org+the-new-hotness to nudge package maintainers to make those updates available to their users).

hroncok commented 6 years ago

pip PR: https://github.com/pypa/pip/pull/5368

pradyunsg commented 6 years ago

Oh, while we're here, who all should I ping when there's something related to pip and how it interacts with Linux Distros?

pradyunsg commented 6 years ago

Maybe @doko42 (if not @warsaw) is the right person to ping for Debian? 1

hroncok commented 6 years ago

For Fedora that would be @torsava @stratakis and me on GitHub. Don't know if you can ping a group with @fedora-python.

pradyunsg commented 6 years ago

Thanks for the info @hroncok!

I don't think you can send notifications by mentioning a group.

mbargull commented 6 years ago

Oh, while we're here, who all should I ping when there's something related to pip and how it interacts with Linux Distros?

Even though there is no "Conda Linux", I'd like to bring up some Conda folks like @kalefranz and @jakirkham, if you don't mind (in reference to assessments like in https://github.com/pypa/pip/issues/5060#issuecomment-377824225).

pradyunsg commented 6 years ago

Sure thing, thanks @mbargull. :)

doko42 commented 6 years ago

On 10.05.2018 15:11, Pradyun Gedam wrote:

Thanks for the info @hroncok!

I don't think you can send notifications by mentioning a group.

I'm not really maintaining python-pip. But if you want to meet up here at PyCon, both Barry and myself are at the venue, including the sprints.

And please, never touch/remove system installed eggs with pip. That's really a no-go. Never. Ever.

Thanks, Matthias

pradyunsg commented 6 years ago

I'm not really maintaining python-pip.

Noted. Is there someone that could be pinged for issues that might affect python-pip? (like #5354)

But if you want to meet up here at PyCon, both Barry and myself are at the venue, including the sprints.

I won't be attending. I believe @dstufft will be in attendance as per https://wiki.python.org/psf/PackagingSprints. :)

And please, never touch/remove system installed eggs with pip. That's really a no-go. Never. Ever.

That'd be #1668. Or is there something I'm not understanding?

pfmoore commented 6 years ago

And please, never touch/remove system installed eggs with pip. That's really a no-go. Never. Ever.

That'd be #1668. Or is there something I'm not understanding?

I presume it's also "don't use sudo pip", which we push as hard as we can, but people still do it :-(

hroncok commented 6 years ago

While we are using this issue to talk about everything distro related... Also meet @mcyprian who has been working on https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe and is currently trying to figure out how to make sudo pip install -U not remove stuff from /usr/ while upgrading into /usr/local/.

pradyunsg commented 6 years ago

Updated the title to be more on-topic to how this discussion went. :P

eli-schwartz commented 6 years ago

The Arch Linux maintainer for pip would be @felixonmars, though I would not be averse to being pinged myself as well.

(I don't have access to the [extra] repository where we package pip, but I do do some python packaging in our [community] repository, and am in general pretty interested, so I could help figure issues out even if @felixonmars makes the final calls etc.)

pradyunsg commented 6 years ago

So, right now we have a change in, so that pip won't print an "upgrade me!" message if the "INSTALLER" of pip is not pip -- #5368.

pradyunsg commented 6 years ago

Another thing we might want to do is what @benoit-pierre suggested -- suppressing the message if pip is unvendored.

eli-schwartz commented 6 years ago

Distros might or might not choose to unvendor pip. I guess a more reliable check is whether the INSTALLER metadata is there. Is there a situation where pip is devendored by the distro but installed using itself?

I cannot think of a reason to install pip using pip for distribution packaging, personally -- it works quite well with python setup.py build && python setup.py install --root="$DESTDIR", in which case there will be no INSTALLER. This is what we do in Arch Linux, certainly. It's also consistent with the general state of packaging python modules for distribution packages for which we use setuptools.

Using pip to install pip would lead to a bootstrap problem, how do you get pip in order to use pip? We currently have no bootstrap issues (I think) other than setuptools, which we use to install everything else (but which depends on packaging and appdirs.)

We try to keep bootstrapping problems to a minimum, because they require additional work to stage temporary bootstrap-only packages that let the build proceed.

pradyunsg commented 6 years ago

5603 documents the changes discussed here, in the vendoring/debundling documentation.

nehaljwani commented 6 years ago

Why not go one step further and also disable upgrades for packages which were not installed via pip? For example, this is very much possible:

$ rpm -qf /usr/lib64/python2.7/site-packages/nacl
python2-pynacl-1.2.0-2.fc28.x86_64

$ pip freeze
cffi==1.11.5
ply==3.9
pycparser==2.14
PyNaCl==1.2.0
six==1.11.0

$ sudo pip install --upgrade pynacl
...
Successfully installed pycparser-2.18 pynacl-1.2.1

pradyunsg commented 6 years ago

@nehaljwani That's something that we can discuss in a new feature request issue. :)

pradyunsg commented 6 years ago

Thanks for filing #5605; if anyone from this thread wants to chip in over there, they're welcome to. :)

pradyunsg commented 5 years ago

I'm going to go ahead and close this -- this issue still has the current reference for @pypa/pip-committers, on who to ping if there's an issue regarding Linux vendors.

ncoghlan commented 5 years ago

New issue to start discussing a general purpose _platformhints module to let Python level installers issue queries to the underlying platform: https://github.com/pypa/pip/issues/6468

pradyunsg commented 5 years ago

I'm going to go ahead and close this

I'm slow at clicking buttons. :)

pypa / pip

Upgrade Message on Linux (and talking to Linux Vendors) #5346

5603 documents the changes discussed here, in the vendoring/debundling documentation.