Docker Compose V1 causes the Install Python packages step to crash

[synweap15]

pip install docker-compose causes an error when running on a fresh build Python 3.10, Python 3.11. Docker Compose V1 is no longer maintained as of July 2023: https://docs.docker.com/compose/

pip install docker-compose
Defaulting to user installation because normal site-packages is not writeable
Collecting docker-compose
  Downloading docker_compose-1.29.2-py2.py3-none-any.whl (114 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.8/114.8 KB 2.6 MB/s eta 0:00:00
Requirement already satisfied: requests<3,>=2.20.0 in /usr/lib/python3/dist-packages (from docker-compose) (2.25.1)
Collecting websocket-client<1,>=0.32.0
  Downloading websocket_client-0.59.0-py2.py3-none-any.whl (67 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.2/67.2 KB 11.0 MB/s eta 0:00:00
Collecting dockerpty<1,>=0.4.1
  Downloading dockerpty-0.4.1.tar.gz (13 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: jsonschema<4,>=2.5.1 in /usr/lib/python3/dist-packages (from docker-compose) (3.2.0)
Collecting docker[ssh]>=5
  Downloading docker-6.1.3-py3-none-any.whl (148 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 148.1/148.1 KB 23.9 MB/s eta 0:00:00
Collecting texttable<2,>=0.9.0
  Downloading texttable-1.6.7-py2.py3-none-any.whl (10 kB)
Collecting docopt<1,>=0.6.1
  Downloading docopt-0.6.2.tar.gz (25 kB)
  Preparing metadata (setup.py) ... done
Collecting python-dotenv<1,>=0.13.0
  Downloading python_dotenv-0.21.1-py3-none-any.whl (19 kB)
Collecting PyYAML<6,>=3.10
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 KB 24.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [43 lines of output]
      running egg_info
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
          return self._get_build_requires(
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 143, in _get_build_requires
          self.run_setup()
        File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 271, in <module>
          setup(
        File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 148, in setup
          return run_commands(dist)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/core.py", line 163, in run_commands
          dist.run_commands()
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 967, in run_commands
          self.run_command(cmd)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/dist.py", line 986, in run_command
          cmd_obj.run()
        File "/usr/lib/python3/dist-packages/setuptools/command/egg_info.py", line 299, in run
          self.find_sources()
        File "/usr/lib/python3/dist-packages/setuptools/command/egg_info.py", line 306, in find_sources
          mm.run()
        File "/usr/lib/python3/dist-packages/setuptools/command/egg_info.py", line 541, in run
          self.add_defaults()
        File "/usr/lib/python3/dist-packages/setuptools/command/egg_info.py", line 578, in add_defaults
          sdist.add_defaults(self)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/sdist.py", line 228, in add_defaults
          self._add_defaults_ext()
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/command/sdist.py", line 312, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
        File "setup.py", line 201, in get_source_files
          self.cython_sources(ext.sources, ext)
        File "/usr/lib/python3/dist-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

More info: https://stackoverflow.com/questions/76868274/build-failed-with-aws-ebcli-on-python-3-11-4

For this role, settings docker__pip_docker_compose_state: "absent" solves it, and I think it should be the default?

Thanks @nickjj for a great plugin!

[nickjj]

Hi,

What distro and CPU architecture are you using? Also how did you install Python? This failure looks like PyYAML isn't able to be compiled and there's no wheel available to avoid compiling dependencies.

But yes, separate to that defaulting to v2 would be a good idea.

[synweap15]

Hello @nickjj,

It's a droplet on DigitalOcean.

Distro: Ubuntu 22.04.2 LTS Arch: x86_64

Python preinstalled with version 3.10, installed 3.11 with ppa:deadsnakes as an alternative.

Indeed, PyYAML seems to be to blame here, but since Docker Compose v1 is no longer maintained, there does not seem to be a sane way to continue supporting the old release.

Thank you @nickjj!

[nickjj]

Version v2.3.0 no longer installs Docker Compose v1 by default. That's available to install from the Galaxy.

As for your issue with PyYAML, I'm not able to reproduce it on a system that has python3 installed. I tried a bunch of assorted Ubuntu and Debian releases on multiple cloud providers and a vanilla Ubuntu ISO. It worked in every case.

[kunaljha5]

You can reproduce this issue on Ubuntu 24.04 with Python 3.12. Since upgrading the docker-compose version isn't feasible within a day, you will need to use version 1.29.2. This is the final Compose V1 release, dated May 10, 2021, and it has not received any security updates since. Proceed with caution.

Manual intervention is required; otherwise, the role may need to be updated.

To resolve the issue, execute the following commands on the remote machine:

# Install the Python 3.12 virtual environment package
apt install python3.12-venv

# Create a virtual environment
python3 -m venv /usr/local/lib/docker/virtualenv

# Activate the virtual environment
source /usr/local/lib/docker/virtualenv/bin/activate

# Install pyyaml version 5.3.1
pip3 install -U pyyaml==5.3.1

By performing these steps, you can install pyyaml version 5.3.1 within the virtual environment. Docker Compose V1 (docker-compose 1.29.2) requires at least pyyaml version 5.3.1. Resolving this dependency manually will fix the Docker Compose installation process.

Successful Logs.

TASK [nickjj.docker : Symlink selected Python package binaries to /usr/local/bin] **************************************************************
skipping: [bot-01] => (item={'name': 'docker', 'state': 'present'})
skipping: [bot-02] => (item={'name': 'docker', 'state': 'present'})
skipping: [bot-03] => (item={'name': 'docker', 'state': 'present'})
changed: [bot-02] => (item={'name': 'docker-compose', 'version': '1.29.2', 'path': '/usr/local/bin/docker-compose', 'src': '/usr/local/lib/docker/virtualenv/bin/docker-compose', 'state': 'present'})
changed: [bot-01] => (item={'name': 'docker-compose', 'version': '1.29.2', 'path': '/usr/local/bin/docker-compose', 'src': '/usr/local/lib/docker/virtualenv/bin/docker-compose', 'state': 'present'})
changed: [bot-03] => (item={'name': 'docker-compose', 'version': '1.29.2', 'path': '/usr/local/bin/docker-compose', 'src': '/usr/local/lib/docker/virtualenv/bin/docker-compose', 'state': 'present'})

@nickjj

[nickjj]

Is the issue here that pyyaml needs to be locked to 5.3.1? This role already sets up a virtual environment.

[akamch]

Is the issue here that pyyaml needs to be locked to 5.3.1? This role already sets up a virtual environment.

Essentially, yes. Seems like it's either 1) don't install Docker Compose v1 (which is the role's default behaviour) or 2) lock pyyaml to 5.3.1.

[nickjj]

The role supports defining docker__pip_packages: [], you could do something like this:

docker__pip_packages:
  - name: "pyyaml"
    version: "5.3.1"

It's not ideal, but if we really wanted to continue supporting Docker Compose v1 that could be moved to docker__default_pip_packages so it's installed by default but only if Docker Compose v1 is enabled. I'm not sure if it's worth optimizing or complicating the role to support something that has been deprecated for 3+ years.

[akamch]

The role supports defining docker__pip_packages: []...

That is exactly what we've been doing at our place for several weeks now. Indeed, it would have been nice to have pyyaml 5.3.1 locked. We transitioned to Docker Compose v2 a couple months ago (but left DCv1 for compatibility), so now it looks like the best course of action is to make the last push and remove DCv1 and docker__pip_packages workarounds. Thus, I have no say on how to make this palatable for those who would like to continue using DCv1.

[nickjj]

Yep, I would suggest removing v1 at this point if everything is confirmed to be working with v2. If other issues like this pop up for others we can consider adding it by default but for now maybe it's worth sticking with it as is.