[BUG] docker-compose - current module uses now depreciated python library. should be migrated to use docker-compose v2 or docker compose commands

ITJamie commented 2 years ago

Description the current docker-compose integration uses the old docker-compose v1 python library, this is now Deprecated. docker-compose v2 is go based. since its creation there is now also a docker compose command in newer docker versions.

As of oct 22 there will be no more security fixes for the v1 branch

A plan to eventually move to ether v2 docker-compose or supporting the in-built compose command in docker should be considered

Additional context https://docs.docker.com/compose/#compose-v2-and-the-new-docker-compose-command https://www.docker.com/blog/announcing-compose-v2-general-availability/ https://github.com/docker/roadmap/issues/257

module: https://docs.saltproject.io/en/latest/ref/modules/all/salt.modules.dockercompose.html

ITJamie commented 2 years ago

A python library alternative to consider. https://github.com/gabrieldemarmiesse/python-on-whales

It was mentioned on dockers own blog https://www.docker.com/blog/guest-post-calling-the-docker-cli-from-python-with-python-on-whales/

davama commented 2 years ago

Was just testing this module (which didnt work). Glad this ticket is here :+1:

Thanks!

ITJamie commented 2 years ago

@Ch3LL i think i can get at least a v1 of a pr for this. Moving from using docker-compose python library to python-on-whales.

If this is something salt would consider what kind of deprecation warning would need to be made in the upcoming release? (This will be a similar to the recent slack engine change which was also a major rewrite and python library change)

Ch3LL commented 2 years ago

I'm for migrating to this new library. We would need to make sure we add a deprecation path for the old library adn support both for those couple of releases..

Let me check with others on my team that they are okay with this as well.

ping @garethgreenaway and @whytewolf any thoughts here?

garethgreenaway commented 2 years ago

No objections.

ITJamie commented 2 years ago

Ive put together https://github.com/saltstack/salt/pull/63051 which covers all the main docker compose functions using python on whales. There is a non-docker standard commands that have been custom written that are a bit... odd.

"create" is just creating and validating a docker-compose file. there is actually a "create" function in docker compose that creates the containers (but does not start them). this feels very wrong to use the "create" term but not actually use docker composes "create" command.
"service_create", "service_upsert", "service_remove", "service_set_tag" - these are custom pieces of code that are making changes to the docker-compose yml file. some dont need adjusment, some do.

ITJamie commented 1 year ago

Can i suggest that as part of the move to python-on-whales we adjust the creat command to actually run the create command with docker compose. That way the functions are 1-1 mapped to the same cli commands in docker compose. @Ch3LL

Ch3LL commented 1 year ago

I'm not familiar enough with that module to say yes or no. I think the most important thing is that functionality and usage does not change for users

Ch3LL commented 1 year ago

I'll give your PR a review when its not in draft anymore. Feel free to ping me on the PR when its ready

Schachte commented 1 year ago

Is there an update on compose usage with latest docker/docker-compose and salt?

Plagued with:

 Reason: 'dockercompose.up' is not available.

despite having both docker and docker-compose installed via pip and present on the minion.

ingjirikratochvil commented 1 year ago

If you are using salt one-dir, you have to install the docker-compose pip package via salt, rather than directly invoking the pip command in shell.

I was getting this error

salt-master:~# salt docker-minion dockercompose.up /docker
docker-minion:
    'dockercompose' __virtual__ returned False: The dockercompose execution module not loaded: compose python library not available.

It was because the docker-compose package was not installed (in one-dir)

salt-master:~# salt docker-minion pip.list | grep -i docker
docker-minion:
salt-master:~#

It showed as installed when I ran pip list in shell directly on the minion, however.

docker-minion:~# pip list | grep -i docker
docker             6.1.3
docker-compose     1.29.2
dockerpty          0.4.1

After I installed it (in one-dir).

salt-master:~# salt docker-minion pip.install docker-compose
docker-minion:
    ----------
    pid:
        15614
    retcode:
        0
...

salt-master:~# salt docker-minion pip.list | grep -i docker
docker-minion:
    docker:
    docker-compose:
    dockerpty:

It started working

I am on salt version 3006.1

primechuck commented 1 year ago

I was looking to add testing to the PR and update it to support more docker compose specific features like multiple files, but the more I looked at it, the more I realized the docker-py doesn't support Docker compose at all. Would it be better to do a rip and replace from docker-py (for salt.dockercompose) to python on whales. docker-py explicitly states they will never support compose. Switching the library would give all the features for compose while leaving the other docker container pieces alone.

ITJamie commented 1 year ago

Ive already prepped a pr for v2 here but i need help getting tests for it https://github.com/saltstack/salt/pull/63051

tovkachua commented 1 year ago

Hello, I'm sorry in advance, if I left my message on the wrong issue. I can't install a docker-compose module on my minion from salt-master. Is this because of the depreciated Python library? Thanks!

(venv) myuser@saltmaster:~/salt-ssh$ salt --versions-report
Salt Version:
          Salt: 3006.4

Python Version:
        Python: 3.10.13 (main, Oct  4 2023, 21:54:22) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.12
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-87-generic
        system: Linux
       version: Ubuntu 22.04.3 jammy

(venv) myuser@saltmaster:~/salt-ssh$ sudo salt 'salt-minion' pip.install docker-compose
[7/1864]hms:
    ----------
    pid:
        10919
    retcode:
        1
    stderr:
          error: subprocess-exited-with-error
          × Getting requirements to build wheel did not run successfully.
          │ exit code: 1
          ╰─> [20 lines of output]
              Error in sitecustomize; set PYTHONVERBOSE for traceback:
              TypeError: expected str, bytes or os.PathLike object, not NoneType
              Fatal Python error: init_import_site: Failed to import the site module
              Python runtime state: initialized
              Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site.py", line 617, in <module>
                  main()
                File "/opt/saltstack/salt/lib/python3.10/site.py", line 610, in main
                  execsitecustomize()
                File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 933, in wrapper
                  import sitecustomize
                File "/tmp/pip-build-env-yw9s_lz0/site/sitecustomize.py", line 23, in <module>
                  site.addsitedir(path)
                File "/opt/saltstack/salt/lib/python3.10/site.py", line 216, in addsitedir
                  sitedir, sitedircase = makepath(sitedir)
                File "/opt/saltstack/salt/lib/python3.10/site.py", line 97, in makepath
                  dir = os.path.join(*paths)
                File "/opt/saltstack/salt/lib/python3.10/posixpath.py", line 76, in join
                  a = os.fspath(a)
              TypeError: expected str, bytes or os.PathLike object, not NoneType
              [end of output]
          note: This error originates from a subprocess, and is likely not a problem with pip.
        error: subprocess-exited-with-error
        × Getting requirements to build wheel did not run successfully.
        │ exit code: 1
        ╰─> See above for output.
        note: This error originates from a subprocess, and is likely not a problem with pip.
    stdout:
        Collecting docker-compose
          Using cached docker_compose-1.29.2-py2.py3-none-any.whl (114 kB)
        Collecting PyYAML<6,>=3.10 (from docker-compose)
          Using cached PyYAML-5.4.1.tar.gz (175 kB)
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
          Getting requirements to build wheel: started
          Getting requirements to build wheel: finished with status 'error'
ERROR: Minions returned with non-zero exit code

cplmayo commented 8 months ago

Ive already prepped a pr for v2 here but i need help getting tests for it #63051

Hitting this issue and based on the age of your PR I wanted to check and see if there was a fix or if I should go down another route.

Tried ng and ran into problems so wanted to go down this route but this is breaking for me. Guessing next alternative is use the formula to install docker then manage through cli and compose commands.

$salt 'node1' pip.install docker-compose
node1:
    ----------
    pid:
        2464
    retcode:
        1
    stderr:
          error: subprocess-exited-with-error

          × Getting requirements to build wheel did not run successfully.
          │ exit code: 1
          ╰─> [20 lines of output]
              Error in sitecustomize; set PYTHONVERBOSE for traceback:
              TypeError: expected str, bytes or os.PathLike object, not NoneType
              Fatal Python error: init_import_site: Failed to import the site module
              Python runtime state: initialized
              Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site.py", line 617, in <module>
                  main()
                File "/opt/saltstack/salt/lib/python3.10/site.py", line 610, in main
                  execsitecustomize()
                File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 932, in wrapper
                  import sitecustomize
                File "/tmp/pip-build-env-6lgzbfx_/site/sitecustomize.py", line 23, in <module>
                  site.addsitedir(path)
                File "/opt/saltstack/salt/lib/python3.10/site.py", line 216, in addsitedir
                  sitedir, sitedircase = makepath(sitedir)
                File "/opt/saltstack/salt/lib/python3.10/site.py", line 97, in makepath
                  dir = os.path.join(*paths)
                File "/opt/saltstack/salt/lib/python3.10/posixpath.py", line 76, in join
                  a = os.fspath(a)
              TypeError: expected str, bytes or os.PathLike object, not NoneType
              [end of output]

          note: This error originates from a subprocess, and is likely not a problem with pip.
        error: subprocess-exited-with-error

        × Getting requirements to build wheel did not run successfully.
        │ exit code: 1
        ╰─> See above for output.

        note: This error originates from a subprocess, and is likely not a problem with pip.

        [notice] A new release of pip is available: 23.3.2 -> 24.0
        [notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
    stdout:
        Collecting docker-compose
          Using cached docker_compose-1.29.2-py2.py3-none-any.whl (114 kB)
        Collecting PyYAML<6,>=3.10 (from docker-compose)
          Using cached PyYAML-5.4.1.tar.gz (175 kB)
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
          Getting requirements to build wheel: started
          Getting requirements to build wheel: finished with status 'error'
ERROR: Minions returned with non-zero exit code

Securitybits-io commented 1 month ago

Running into the same issue as @cplmayo

ITJamie commented 1 month ago

I have a variant here which works: https://github.com/ITJamie/salt/blob/dockercomposev2/salt/modules/dockercomposev2.py

you will need to salt-pip install python-on-whales to install the required library

you can drop this file into your salt_states folder eg states/_modules/dockercomposev2.py and dockercomposev2 module commands will be available

until salt 3008 is out and there is a new home for the salt-docker extension for modules/states this is the best i can do. im happy to pr the above new file into the extension when the new salt extension repo exists for docker

saltstack / salt

[BUG] docker-compose - current module uses now depreciated python library. should be migrated to use docker-compose v2 or docker compose commands #62743