Removing EXTERNALLY-MANAGED flag leads to "non-default" configurations, which could lead to inaccurate tests and confusing DX

[jaydrogers]

Hey Jeff,

Hope you're doing well! I'm beyond grateful for all your work and I have learned a ton. I've been working on an open source project and ran into an issue.

I'm basically opening up this issue as my own notes and research on how to resolve this. If you have time to chime in and share thoughts -- by all means, I'd love to hear your thoughts if you have any.

🤔 Problem

When installing a Docker package with PIP on Ubuntu 24.04, a fatal error displays:

Code:

- name: Ensure 'docker' python package is installed.
  ansible.builtin.pip:
    name: docker
    state: latest

Error:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

🔀 Workaround

The suggested workaround (which is implemented in this image) is to remove the EXTERNALLY-MANAGED flag.

https://github.com/geerlingguy/docker-ubuntu2404-ansible/blob/ef6b0d635f8a859defcd0ca92beaae18ef042fbf/Dockerfile#L31-L32

Jeff talks about why he did this here: https://www.jeffgeerling.com/blog/2023/how-solve-error-externally-managed-environment-when-installing-pip3

😅 Unintended Side Effects

The side effects of this workaround lead to my tests passing:

But when I ran my playbook on a brand new Ubuntu 24.04 server from Hetzner, it failed:

TASK [serversideup.spin.swarm : Ensure 'docker' python package is installed.] ******************************
fatal: [spin.test]: FAILED! => {"changed": false, "cmd": ["/usr/bin/python3", "-m", "pip.__main__", "install", "-U", "docker"], "msg": "\n:stderr: error: externally-managed-environment\n\n× This environment is externally managed\n╰─> To install Python packages system-wide, try apt install\n    python3-xyz, where xyz is the package you are trying to\n    install.\n    \n    If you wish to install a non-Debian-packaged Python package,\n    create a virtual environment using python3 -m venv path/to/venv.\n    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make\n    sure you have python3-full installed.\n    \n    If you wish to install a non-Debian packaged Python application,\n    it may be easiest to use pipx install xyz, which will manage a\n    virtual environment for you. Make sure you have pipx installed.\n    \n    See /usr/share/doc/python3.12/README.venv for more information.\n\nnote: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.\nhint: See PEP 668 for the detailed specification.\n"}

This means that every user that provisions a new Ubuntu 24.04 server will need to also remember to remove this EXTERNALLY-MANAGED file in order to have a quality experience with Pyhton, Ansible, and more.

🙏 Possible Solutions

To resolve this, I'm thinking:

Update my playbook to remove /usr/lib/python3.12/EXTERNALLY-MANAGED

I could mimic these images, but I'm unsure about the longevity of this fix. Especially since the path to python is fixed to a specific version.

Install pipx

Jeff points out using the pipx project, but I am unsure if this will cause python headaches or Ansible package issues with other projects.

There is an Ansible community module for managing PIP dependencies in a virtual environment: https://docs.ansible.com/ansible/latest/collections/community/general/pipx_module.html

Install with apt install python3-docker

Installing via the system python works, but then I may lose my ability to run the latest version of pip docker. This could cause me to be stuck waiting for Ubuntu package repositories being updated if there are important bug fixes.

Next Steps

If anyone has any thoughts on this, I'm all ears to learn more. I'll continue researching this and will update this thread as I learn more.

[geerlingguy]

Unless you need to do a lot of python stuff on a server, I have no qualms about nuking /usr/lib/python3.12/EXTERNALLY-MANAGED on every Ubuntu server I come across.

Basically I hate PEP 668

[jaydrogers]

If you're confident about this, then I might just update my side to remove the file as well.

I only need the systems to install pip install docker, so I don't think it's asking for a lot.

I started playing around with the rabbit hole of pipx and virtual environments, and it turned into a black hole instead of a rabbit hole 😆

Closing this issue for now. I appreciate your insight!

[geerlingguy]

I started playing around with the rabbit hole of pipx and virtual environments, and it turned into a black hole instead of a rabbit hole 😆

That's exactly why I am okay deleting the file and moving on. There are cases where virtual environments are wonderful but I don't want to be forced to use them, because they add complexity that I don't care for :)