search

python / cpython

The Python programming language
https://www.python.org/
Other
60.26k stars 29.15k forks source link

ensurepip has no --break-system-packages option #118384

Open bjourne opened 1 month ago

bjourne commented 1 month ago

Bug report

Bug description:

$ python3 -m ensurepip --upgrade                                                                                                                                                                                   
error: externally-managed-environment
...
You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
$ python3 -m ensurepip --upgrade --break-system-packages
python -m ensurepip: error: unrecognized arguments: --break-system-package

So it needs the --break-system-packages option or the error message should not tell me to use it.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

terryjreedy commented 1 month ago

I suspect ensurepip should not pass this message on from pip.

@pfmoore Are you involved with ensurepip?

pfmoore commented 1 month ago

I think the situation is a little more complex than that, unfortunately. Unless you pass the --break-system-packages flag to pip, it simply won't install - and ensurepip doesn't have a way to set that option as @bjourne pointed out.

However, I'm a little unclear about the scenario here. Normally, ensurepip is intended for use with a Python installation that you built yourself, so unless you marked the Python environment as externally managed, everything will work fine. So this looks as if the OP is running ensurepip on their system Python - and that's exactly the situation that should be blocked, because OS vendors don't want pip messing with the software they manage. If the user wants to install or upgrade pip, they should be using the OS supplied package that contains pip (assuming for some reason that their OS doesn't install pip when installing Python).

So I think I'd want to know more about why the user is running python3 -m ensurepip --upgrade in the first place. @bjourne can you clarify?

bjourne commented 1 month ago

It's beside the point - regardless of how I use Python the error message shouldn't tell me to do something I can't do.

Since I'm unhappy with how my OS vendor has decided to package Python, Haskell, and other language's runtimes I want to manage them myself. This I do by installing pip in ~/.local/bin which works perfectly fine (or at least used to). "We're all consenting adults here" and I want to continue with my workflow.

pfmoore commented 1 month ago

You can use pip (with the --break-system-packages flag). Why do you specifically need to use ensurepip? It's designed for preinstalling pip as part of building Python, not for general use. For your use case I'd suggest downloading get-pip.py and using that to install pip wherever you want.

I agree that the message is a bit misleading - if someone was motivated, they could look at the possibility of creating a PR that suppressed the one line "You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages." from the pip output, simply failing without suggesting that option. But to avoid this becoming a maintenance burden, it would need to be done in a way that didn't require updating whenever pip changed the wording of that message - and that's likely to be more work than it's worth, given that the benefit is relatively minor.

bjourne commented 1 month ago

I'm getting a new error message from python -m ensurepip --upgrade now:

error: externally-managed-environment
...Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.12/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 284, in _main
    return _bootstrap(
           ^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 200, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 101, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpbtsaraow/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpbtsaraow\', \'--upgrade\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

I used ensurepip because the documentation recommends it: "The ensurepip package provides support for bootstrapping the pip installer into an existing Python installation or virtual environment. This bootstrapping approach reflects the fact that pip is an independent project with its own release cycle, and the latest available stable version is bundled with maintenance and feature releases of the CPython reference interpreter.

In most cases, end users of Python shouldn’t need to invoke this module directly (as pip should be bootstrapped by default), but it may be needed if installing pip was skipped when installing Python (or when creating a virtual environment) or after explicitly uninstalling pip." I didn't know about get-pip.py.

pfmoore commented 1 month ago

I asked on the PEP 668 (externally managed environments) discussion thread what the PEP authors thought about this situation: https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/117

bjourne commented 1 month ago

Wow, thanks for the work you do!

eli-schwartz commented 1 week ago

You can use pip (with the --break-system-packages flag). Why do you specifically need to use ensurepip? It's designed for preinstalling pip as part of building Python, not for general use. For your use case I'd suggest downloading get-pip.py and using that to install pip wherever you want.

get-pip.py requires downloading and running some code from the internet which isn't verified or trivially verifiable. And pip doesn't have to be used with pypi.org (or indeed for downloading wheels from the internet) so the fact that it can't cryptographically verify the authenticity of a pypi package before installing it, isn't necessarily a reason to say that it's no different from get-pip.py

Besides: ensurepip is a public, and publicly documented, module. It's not obvious to me that it isn't for general use. It's being used here for precisely its documented purpose.

But neither of these are the case. In fact, https://pip.pypa.io/en/stable/installation/ goes out of its way to list ensurepip as one of the ways to "officially" install pip, in addition to get-pip.py. In other words, pip directly encourages end users to run ensurepip, as an end user tool.

...

I don't think it makes sense to declare ensurepip as something not for general use. And if it's for general use, then it should support the flag it tells you to use to make sure it works.

eli-schwartz commented 1 week ago

Additionally, ensurepip supports the --user flag, which is clearly not intended for use by the preinstallation process, nor is it intended for use by venvs.

Since I'm unhappy with how my OS vendor has decided to package Python, Haskell, and other language's runtimes I want to manage them myself. This I do by installing pip in ~/.local/bin which works perfectly fine (or at least used to). "We're all consenting adults here" and I want to continue with my workflow.

Its sole use is to do precisely as the OP wants: disregard the choices of the OS vendor, and install pip to ~/.local/bin / ~/.local/lib/python3.XX/site-packages.

pfmoore commented 1 week ago

Note that the requirement to use --break-system-packages is also a choice made by the OS vendor, and the OP can override that by deleting the EXTERNALLY-MANAGED file that the vendor added (assuming they understand, and are comfortable with, the implications of doing so).

To be 100% clear, the OP hasn’t said how they built their copy of Python. It seems to me very odd that a user-built Python installed into ~/.local/bin should have a `site-packages’ that points to a system-controlled directory. So honestly, I think the OP has either misconfigured their build, or made some choices without fully understanding the implications. But they have options to do what they want to do, so that should be sufficient.

eli-schwartz commented 1 week ago

I'm not sure how you concluded that python is installed to ~/.local/bin just because the OP is attempting to install pip there.

The OP was very clear that it is not a user-built python.

Since I'm unhappy with how my OS vendor has decided to package Python,

This I do by installing pip in ~/.local/bin

pfmoore commented 1 week ago

Apologies, I misread the comment as saying that they had installed python in ~/.local/bin.

But I'm now even more confused, as I don't see how ensurepip would install pip in the user's ~/.local/bin directory if Python was installed in a system directory, without --user. And I don't see why pip would give the "you must use --break-system-packages" message if installing itself to ~/.local.

The OP originally said:

python3 -m ensurepip --upgrade So it needs the --break-system-packages option or the error message should not tell me to use it.

My response was yes, the message is not ideal and could be changed, but it's non-trivial to do so. If someone wants to raise a PR for that, we can thrash out the details in that PR.

I do not think ensurepip needs the --break-system-packages option, because if the OS vendor has marked the location where pip is installed as system managed (using the EXTERNALLY-MANAGED file) then python3 -m ensurepip --upgrade should fail. If a user wants it to work, they have to override the OS vendor's choice, which they can do by removing the EXTERNALLY-MANAGED file (or by simply using pip install --upgrade --break-system-packages pip). They need to understand the consequences of what they are doing, obviously. We don't support people blindly ignoring their OS vendor's restrictions.

@eli-schwartz do you have an actual, reproducible issue where you are seeing this error, or are you simply arguing principles based on the OP's original report? Because I got the impression that the OP was happy with the original response (@bjourne if you do still have an issue here, please feel free to say so - I don't want the current discussion to distract from making sure you have a solution you are happy with).

eli-schwartz commented 1 week ago

Apologies, I misread the comment as saying that they had installed python in ~/.local/bin.

But I'm now even more confused, as I don't see how ensurepip would install pip in the user's ~/.local/bin directory if Python was installed in a system directory, without --user.

Ensurepip bundles a copy of pip, and injects itself into the path so that it can import pip and use pip to install a pip wheel. Pip itself contains the functionality for automatically assuming --user when you run its installation procedures.

So running /usr/bin/python -m ensurepip as a local user has always installed pip to ~/.local, for the same reason that /usr/bin/python -m pip install -U pip has done so.

This is an important and often forgotten-about safety feature of pip. My personal opinion is that it was prescient, and rendered EXTERNALLY-MANAGED unnecessary, but clearly not everyone agrees...

And I don't see why pip would give the "you must use --break-system-packages" message if installing itself to ~/.local.

Because pip has another implemented functionality, to require --break-system-packages (a flag that claims you're breaking the packages in the global site-packages) even when used with --user to operate on the user site-packages. I think this functionality is unhelpful, but I don't make the rules of pip.

I do not think ensurepip needs the --break-system-packages option, because if the OS vendor has marked the location where pip is installed as system managed (using the EXTERNALLY-MANAGED file) then python3 -m ensurepip --upgrade should fail. If a user wants it to work, they have to override the OS vendor's choice, which they can do by removing the EXTERNALLY-MANAGED file (or by simply using pip install --upgrade --break-system-packages pip). They need to understand the consequences of what they are doing, obviously. We don't support people blindly ignoring their OS vendor's restrictions.

s/pip is installed/pip should get installed/

That argument equally applies to pip. If the OS vendor has marked the location (that is to say, has marked ~/.local/bin by placing the EXTERNALLY-MANAGED file in /usr/lib on the same machine) where pip should be installed, python -m pip install --user requests should fail. If a user wants it to work, they have to override the OS vendor's choice.

So why does pip have an option to override the vendor's choice, but ensurepip doesn't?

@eli-schwartz do you have an actual, reproducible issue where you are seeing this error, or are you simply arguing principles based on the OP's original report?

I'm seeing the exact same error as the OP.

I tried running python -m ensurepip --user and got the "externally managed" error, and the suggestion to use --break-system-packages.

Yes, I went out of my way to verify it again with --user passed to ensurepip, even though it's the default pip functionality.

Because I got the impression that the OP was happy with the original response (@bjourne if you do still have an issue here, please feel free to say so - I don't want the current discussion to distract from making sure you have a solution you are happy with).

The OP responded to you saying you've asked on the python help forum, what others think -- by saying thanks for all the work you do.

I don't see anything there to suggest "happy with the original response" -- quite the contrary, my implicit understanding was that the OP is happy with your willingness to reconsider the matter by asking for further input from the PEP authors, and is awaiting a status update.

bjourne commented 1 week ago

Apologies, I misread the comment as saying that they had installed python in ~/.local/bin.

I have not.

But I'm now even more confused, as I don't see how ensurepip would install pip in the user's ~/.local/bin directory if Python was installed in a system directory, without --user. And I don't see why pip would give the "you must use --break-system-packages" message if installing itself to ~/.local.

For pip, --user is not required to install in ~/.local so I didn't think it would be required for ensurepip either. Regardless --user doesn't work either.

I do not think ensurepip needs the --break-system-packages option, because if the OS vendor has marked the location where pip is installed as system managed (using the EXTERNALLY-MANAGED file) then python3 -m ensurepip --upgrade should fail. If a user wants it to work, they have to override the OS vendor's choice, which they can do by removing the EXTERNALLY-MANAGED file (or by simply using pip install --upgrade --break-system-packages pip).

pip install --upgrade --break-system-packages pip <- impossible since pip is not installed. I didn't know about the EXTERNALLY-MANAGED file, but I don't want to remove it since Arch Linux's package manager will recreate it sooner or later and if I hadn't root I couldn't remove the file.

They need to understand the consequences of what they are doing, obviously. We don't support people blindly ignoring their OS vendor's restrictions.

Fair but --break-system-packages is explicit. I'm not blindingly doing it.

@bjourne if you do still have an issue here, please feel free to say so

No, get-pip.py solved it.

Note: I prefer it like this because for students with no root access and limited home directories installing python packages in ~/.local is the easiest. If they run into issues they can safely rm -rf ~/.local/lib/python3.12 and try again.

pfmoore commented 1 week ago

So running /usr/bin/python -m ensurepip as a local user has always installed pip to ~/.local, for the same reason that /usr/bin/python -m pip install -U pip has done so.

When not run as the owner of the Python installation. It's worth remembering that the ensurepip documentation says

In most cases, end users of Python shouldn’t need to invoke this module directly

But I don't want to nitpick here. Docs can be changed, what matters is making sure people understand the right way to use the module, and don't have unreasonable expectations.

Because pip has another implemented functionality, to require --break-system-packages (a flag that claims you're breaking the packages in the global site-packages) even when used with --user to operate on the user site-packages. I think this functionality is unhelpful, but I don't make the rules of pip.

My apologies, I'd forgotten that subtlety. The reason is the same as all aspects of PEP 668 - installing to user-site can shadow critical system functionality, and distributors, not unreasonably, want to be able to restrict the chance of that happening because a user didn't understand the implications of what they were doing. You're welcome to disagree with that reasoning (as a Windows user, I can't say I like the way Linux distros do things like this, but that's just my viewpoint) but it is what was agreed in the PEP.

So why does pip have an option to override the vendor's choice, but ensurepip doesn't?

Because (as I understand it) ensurepip is intended to be used by the people managing the Python installation, not by users of that installation.

I'm seeing the exact same error as the OP.

OK, and my response is the same: yes, the message is not ideal and could be changed, but it's non-trivial to do so. If someone wants to raise a PR for that, we can thrash out the details in that PR.

my implicit understanding was that the OP is happy with your willingness to reconsider the matter by asking for further input from the PEP authors, and is awaiting a status update.

Feel free to check the linked discussion, and participate there if you want. So far, the only response has been from Debian, who basically said they only use ensurepip to seed venvs, and they direct users to install pip via the Debian-managed pip package. No-one has yet said that they feel users should be running ensurepip on Linux themselves.

I don't think there's anything further to say at the moment. The message can be changed if anyone is interested in putting in the work to do so. The docs can be changed to more explicitly discourage this sort of use of ensurepip, if anyone thinks that would be useful[^1]. I don't see the value of, and won't support, exposing --break-system-packages in ensurepip, although if you write a PR and can get another core dev to approve it, I won't block it (beyond linking back to this discussion for contect).

If the discussion thread results in a different consensus, I'll report back here and we can proceed based on whatever that consensus is.

[^1]: If I were proposing a docs change, I would add a paragraph to say "The ensurepip module is intended for internal use by Python and Python installers. End users should not normally need to use it, it is documented for use by people building or maintaining their own Python distribution. In particular, it should not be used to modify a system-packaged Python installation."