externally-managed-environment on pip3 installation in make apply

[TiiFuchs]

When running make apply something tries to install pip3 packages (I think docker and docker-compose).

But this fails on my Debian 12 system with the following error:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

I could install python3-docker system-wide but not python3-docker-compose (it does not exist).

I tried creating a virtualenv, but the playbook is using directly /usr/bin/python3 so it doesn't matter.

I tried using pipx, but I get the following error, when trying to install pipx install docker-compose:

Fatal error from pip prevented installation. Full pip output in file:
    /root/.local/pipx/logs/cmd_2023-09-14_19.06.36_pip_errors.log

pip seemed to fail to build package:
    PyYAML<6,>=3.10

Some possibly relevant errors from pip install:
    error: subprocess-exited-with-error
    AttributeError: cython_sources

Error installing docker-compose.

I'm lost... Not sure what to do...

[TiiFuchs]

Found a workaround by installing it manually.

apt install python3-docker python3-compose does the trick.

I didn't know that it's python3-compose and not python3-docker-compose...

[TiiFuchs]

Oh no I have to paddle back...

It does not work, I got confused because it broke something else that runs earlier in the runbook.

Installing those manually results in this error:

TASK [docker : Ensure docker-compose symlink.] ****************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "msg": "refusing to convert from file to symlink for /usr/bin/docker-compose", "owner": "root", "path": "/usr/bin/docker-compose", "size": 996, "state": "file", "uid": 0}

I "fixed" that by moving the python script /usr/bin/docker-compose to /usr/bin/docker-compose.bak. Then it creates a symlink and gets back to the error in the first post.

[TiiFuchs]

But I'm actually confused why the error still appears.

Running (like the runbook is doing it) /usr/bin/python3 -m pip list lists the docker as well as the docker-compose package...

It seems that there is no check. It just tries to install it resulting in python3 to error with the message that you should now install global packages that way... (that is so annoying...)

[TiiFuchs]

I worked around that error by removing the corresponding step from the playbook. The packages are there after installing it via apt like described before and can be used successfully. Only the check seems to fail. Removing the step from the playbook results in everything working fine.

[ahembree]

Which task is it failing on? There's a few places where pip is called so I want to make sure we're looking at the same thing. I hope to have some time later this week to troubleshoot.

[TiiFuchs]

The task that checks and tries to install docker and docker-compose via pip

[ahembree]

I've created a new branch (https://github.com/ahembree/ansible-hms-docker/tree/migrate-to-ansible-galaxy-docker-role) to start migrating to a popular Ansible Galaxy role in order to install docker and the pip packages that I believe resolves this.

[ahembree]

I'm still encountering the same issue with the Ansible galaxy role, so this might be an issue with the newer Debian release

[ahembree]

I've completed the migration to an Ansible Galaxy Docker role, so if there any further issues with the installation of Docker, it is unfortunately out of my control.

The Galaxy role will most likely have more support, better features, and stay more up to date. At nearly 17 million downloads, I believe this role is the better option than my custom one: https://galaxy.ansible.com/ui/standalone/roles/geerlingguy/docker/

However, if you experience any issues (other than the externally managed environment issue), let me know and I can investigate and see if changing some variables around will resolve it.

[KnuffigerMaxi]

its still doesnt work...

[ahembree]

@KnuffigerMaxi please submit a new issue with more information on what is not working as this issue is closed. Please include your OS and its version, the Ansible version, and the docker version.