[Docker Compose] V1 End of Life Policy

[stephanierifai]

Tell us about your request End maintenance of Docker Compose v1

Which service(s) is this request for? Dev-tools

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? The goal of compose v2 was to align on Go for most of our development. In order to focus our time towards new features and enhancements, we need to make v2 the default and phase out v1.

Are you currently working around the issue? N/A

Additional context On April 26, 2022, we announced the GA of Docker Compose V2. We want you to have ample time to transition to Compose V2. We won’t sunset Docker Compose V1 immediately, and developers can still revert to V1. Given the numerous successful transitions to Compose V2 so far, we’ve created the following proposed timeline for Docker Compose V1's end of life (EOL):

October 2022 - 6 Months Post GA

• Support of critical bug fixes and severe security issues will end on Compose v1
• Users can alias docker-compose to docker compose
• Users can opt-out of V2 via the Docker Desktop UI or through the docker-compose disable-v2 command

April 2023 - 1 Year Post GA

• Users can alias docker-compose to docker compose
• Users can no longer opt-out of V2 via the Docker Desktop UI or through the docker-compose disable-v2 command in new versions

Note: We have no plans of removing any aliasing of docker-compose to docker compose, we want to make it as easy as possible to switch and not break any of your code.

We’ll monitor feedback here alongside V2’s usage and make adjustments accordingly.

[ndeloof]

proposed notice on project's github: https://github.com/docker/compose/pull/8589

[rfay]

My first response is that this is way too fast, especially since docker-compose v2 is not even yet available for Linux, nor even thought through how it will be available. I think stopping all but security fixes is one thing, but security and basic maintenance probably needs to be a year after v2 becomes available on Linux. And that, of course, depends on how it's made available.

[ndeloof]

not even yet available for Linux

Not yet "as a system package" (work in progress), but definitively available. This is just a question of running a curl command and many already did.

[rfay]

Just saying... as a highly motivated user (I need it for ddev tests on Linux) I've been following all the issues on docker-compose v2. And even I haven't yet discovered how you do it, with or without packaging. I was delighted to see https://github.com/docker/compose-switch come out... but it still has no instructions in the readme. When there's a standard, documented way to use and maintain/upgrade docker-compose v2 on Linux (not docker compose) then it has arrived and the start time for docker-compose on Linux will have arrived.

I do think there's some crosstalk about the availability of the compose plugin for docker, which is mostly do-able on some platforms. But it's not a drop-in replacement or switch for docker-compose. Way too many tools in the linux ecosystem are dependent on docker-compose to hope that docker compose will replace them, even when it becomes easier to obtain.

[ndeloof]

There's no "docker-compose v2 on Linux". Compose V2 is docker compose. We indeed provide a command line compatibility tool, but that's not expected to be the default usage long terms.

About Linux availability, we indeed don't yet offer those components as system packages. Doesn't mean Linux users can't install Compose v2 and compose-switch. We miss documentation for sure, but this is just a question of days to get this available.

[rfay]

I think there may just be a whole bunch of bundled things going on here, and it's going to be important to sort them all out.

1. For some reason, there will not be a docker-compose v2. I don't think users will understand that choice, but if that's what's going on it needs to be stated more clearly.
2. In that case, this isn't about "v1 end of life policy", it's about "docker-compose end of life" So the world needs to know "docker-compose is to be no more"..

There are loads of implications here:

1. The compose plugin needs to be bundled with the docker client everywhere on Linux. For starters, that means the Docker repo packages.
2. The compose plugin needs to make it into the OS-level distributions as soon as possible. It's already missed Debian 11, but could make it into Ubunto 22.04.

I should note that "running curl commands" is not a generally accepted practice for managed Linux machines, so it will be important to get these tools packaged fully as soon as possible.

I think I won't be the only one confused by the decision to do away with docker-compose. It's an old friend with a thousand uses and continuing it as a wrapper on compose v2 would be so terribly easy.

[stephanierifai]

Based on feedback we have decided to focus our efforts on making the experience better for Linux users, this is has been edited above.

For history, the previous proposal was:

🔏 As of September 28th, 2021 new features and bug fixes will only be considered in the V2 codebase. Users will be defaulted into Docker Compose V2 and it will be considered GA. Users can still opt out through the UI and the cli. Docker Compose V1 will only be maintained regarding security issues. 🔐 On December 28th, 2021 Docker Compose V1 will be marked as deprecated. 🔒 By March 28th, 2022 no new contribution will be accepted to the V1 branch, even for security fixes.

[BigMorty]

For how long will users be able to switch back to using docker-compose (v1)?

[jamshid]

Very glad a docker-compose alias will be provided on Linux, we can't rewrite all scripts to use docker compose immediately especially when that requires a not-yet-released docker-ce package on Linux.

Would be nice if docker-compose were automatically installed by yum/apt-get install docker-ce-cli but probably not a big deal, afaik docker-compose was always a separate curl download or pip install.

A little nervous about the push to V2 when there's still some open issues like https://github.com/docker/compose/issues/8505 preventing me from using it fully but hopefully all resolved soon. Also, docker compose up seems to be restarting services unnecessarily (more than V1 docker-compose) but I need to repro and file an issue.

[jpic]

Thank you very much for the new release! It would have been great to keep the convert command or to document that it's been dropped in the transitioning documentation. docker/compose#8914 Too bad for the "feature" I relied on! But the new spinner is nice, I thought compose was using an external library for that but it turns out it's actually implemented in compose codebase

[stephanierifai]

Hi @jpic ! Thanks for your feedback :)

Do you mind sharing what version you are on 🤔 It should be available in V2 https://docs.docker.com/engine/reference/commandline/compose_convert/, but happy to look into it further.

cc @ulyssessouza

[jpic]

Hi @stephanierifai,

The new convert command as implemented in v2 serves a completely different purpose, which is now to "output information duplicated from docker inspect for debugging" (according to this comment), in v1 it used to "output a usable stack configuration", which you could use to spawn new stacks.

The documentation states:

it merges the Compose files set by -f flags, resolves variables in Compose file, and expands short-notation into fully defined Compose model

While it does that, it previously did it so that you could reuse the output to spawn a new stack, and it's not possible anymore with the new output, as documented in issue docker/compose#8914. It also used to generate a more readable format, which it doesn't anymore because it now "expands short-notation into fully defined Compose model", I'm not sure if that's a feature anymore or rather a side effect caused by the current implementation in v2.

The other problem concerning the migration, is that it's not clear whether we should reimplement this feature from v1 on our own, or if compose is going to fix it in further versions ... A clear statement would help a lot!

[ndeloof]

That's both a side effect (go being a typed language, compose config output actually reflect the structs, so the "expand short notation") and a feature: compose config helps diagnose configuration issues as it dump the actual model being applied. Due to the first reason, there's no simple way to implement config command in a comparable way it was done by compose-V1 (i.e. just merge yaml files)

Also, compose config output, despite being verbose, is fully usable stack configuration

[jpic]

Also, compose config output, despite being verbose, is fully usable stack configuration

I'm sorry, does it still output hardcoded full network names with the project name prefix ?

[ndeloof]

It indeed resolves resource names as they will be created, so you can't use to run a distinct stack. The output is a fully usable configuration, but not a configuration you can tweak for another project

[jpic]

If my compose defines a network "foo", then running convert should output it as "foo", not as "directoryname_foo", that's actually modifying the data I'm feeding into convert, it's not the same feature. It should not be named convert. Call that "inspect" if you want, and then decide if you're going to port the convert feature from v1 to v2 or drop it.

[ndeloof]

the command name as documented by the command line is config. convert name comes from earlier work on ECS integration, and has been kept for compatibility.

Whatever the command name is, replicating compose-v1 config command is just not possible with the compose-go library design.

[jpic]

To allow smooth migration from docker-compose, this subcommand declares alias docker compose config

It is a mistake to use the same name for a commands that is not compatible, even more "for compatibility". That causes more migration problems than if the alias just wasn't there at all! It's confusing...

Implementing config in go is possible with a library (mergo) that one of your libraries is using, you could also use a template! Or even do a pass on the data prior to outputting so that the network and volume names you feed into it don't change... Anyway, it'll have to be more than a one liner that is basically the current implementation, but we already have 3 solutions right there.

If you're committing to dropping the feature it's fine, just document it, that's what I'm saying, and we'll maintain it in the python version, because in the mean time it's not clear how we should deal with this.

[ndeloof]

mergo used in compose-go merges go structs after the individual yaml files have been parsed. compose V1 used python dictionaries and merged them by key, before applying validation. So we never will fully reproduce compose V1 config command.

About resources prefix, if this is the only blocker, I guess we could look into being able to not append those in compose-go, so that docker compose config --to-be-defined-flag would render the "reusable output" you expect

[thediveo]

Can't you first merge at the YAML generic model level ("dict" or map level), instead of the unmarshalled Go struct level? Wouldn't that keep merging compatibility with v1 behavior?

[jpic]

Prefixes are one completely unexpected breakage for sure, but the expansion of volumes just seems completely useless.

Feeding:

    volumes:
      - ./syncthing:/var/syncthing
      - ./data/media:/app/media
      - ./spooler:/spooler
      - ./log:/app/log

You get:

volumes:
    - bind:
        create_host_path: true
      source: ./log
      target: /app/log
      type: bind
    - bind:
        create_host_path: true
      source: ./data/media
      target: /app/media
      type: bind
    - bind:
        create_host_path: true
      source: ./spooler
      target: /spooler
      type: bind
    - target: /tmp
      type: volume
      volume: {}
    - bind:
        create_host_path: true
      source: ./syncthing
      target: /var/syncthing
      type: bind

Add random flags to do half of the job if you want, but again I'm not requesting anything, I was just reporting bugs in this new implementation as I naively thought it was just a matter of time before they were fixed, this won't happen ok, I'm fine implementing my own in Python, I'm just now saying this should be documented to help users transitioning from v1 to v2 because the situation with this command is extremely confusing.

It's a mistake to keep the name of the command because this confuses users into thinking that compose v2 is going to try to reimplement it, instead of working around what seems like a broken implementation is fixed they should be re implementing on their own. It's a big difference when you know if your fixes are going to be temporary or permanent.

[ndeloof]

@jpic resource prefix incompatibility has been fixed expanding volumes is important for diagnostic, and doesn't prevent the output to be reused by other compose-related tooling

Could we please stop focussing on tiny implementation details for a specific sub-command that is just a topic for an issue, while we are discussing here the global strategy for Compose V1 End of Life?

[jpic]

Nice! Thank you for fixing the prefix! Volume expansion is absolutely pointless in the example I've given because the content of the bloat lines is exactly the same for every volume, plus there's a new default tmp volume that your config is adding, it will stay as-is even when the upstream default changes, but I understand it's not easy to spot in the bloat.

I'm just backing my suggestion with concrete examples but I'm actually talking about the strategy for v1 EOL: I think documentation should state that the config command has been dropped and that a new similar command may be used instead so at least users don't naively believe that a port of the command is in progress. If it's not going to be the same it should not have the same name because it's confusing.

It's a common practice to document the BC breaks when releasing new versions, I think compose should also be doing that, like any other software... Because it helps transitioning.

[ndeloof]

none of the commands are strictly identical to the v1 one. Event compose version has a distinct output and some might get their script broken by this. This is what a "major version bump" implies, should we add warnings everywhere?

[jpic]

Event compose version has a distinct output and some might get their script broken by this.

Yet it has never been the case and I've had this for years.

This is what I read in the IMPORTANT section of the transitioning documentation as it is today:

We would like to make the Compose V2 transition to be as smooth as possible for all users. We currently don’t have a concrete timeline to deprecate Compose V1. We will review the feedback from the community on the GA and the adoption on Linux, and come up with a plan to deprecate Compose V1. We are not planning to remove the aliasing of docker-compose to docker compose. We would like to make it easier for users to switch to V2 without breaking any existing scripts. We will follow up with a blog post with more information on the exact timeline on V1 deprecation and the end of support policies for security issues.

Nothing could be further from the truth, given your own opinion here as a docker core dev:

This is what a "major version bump" implies

Look Nicolas, there's nothing to be ashamed of. When I do a major bump, I read the release notes which I expect to list the upgrades I'm supposed to do. Here, I've read the "transitioning documentation", and was completely misled.

should we add warnings everywhere?

Not "warnings everywhere" :facepalm: but having a CHANGELOG or writing release notes to describe the BC breaks in new versions is the basics of what we call "maintaining a project", every major release of every project documents BC breaks somewhere, examples are legion:

• https://docs.djangoproject.com/en/dev/releases/4.0/
• https://aws.github.io/aws-sdk-go-v2/docs/migrating/
• https://go.dev/doc/go1.17#language

We're not talking about a little change in a command here, we're talking about a command that was dropped, that now has an alias to the command that was dropped and that doesn't even serve the same purpose at all.

But if you want users to keep doing the effort of naively reporting what they believe are bugs you are not going to fix, then don't change the "transitioning documentation" because it's perfect as it is. Now that I know I just won't report bugs with v2 anymore, but there will be others like me because that's where the current "transitioning documentation" takes us.

[ndeloof]

ok I'm giving up trying to convince you a 2.0 bump should not be expected to be fully transparent, whenever we try to make it as smooth as possible. Up to Docker Inc. to deal with Compose v1 maintenance if they want to :P

[thediveo]

knowing of an industrial project on top of docker-compose v1 this means a complete new integration. At least I would have expected a useful list of changes and not just a sloppy "well, expect things to break for a major semver change". So in the spirit of the initial question: if you cannot make it compatible, then document the many things you change.

[jpic]

If you don't want to document your changes in any version, minor or major, because "users should expect so", then this means your project is stuck in "alpha version" status. It's fine, but it should also be documented because we were used to a "stable" compose.

[stephanierifai]

Hey @jpic, this is really great feedback and we appreciate you raising it for us. The team is actually currently in progress of writing V1 -> V2 migration docs, so I'll certainly raise this as something to include. Happy to chat some more about it or other ideas on how we can make things clearer. If you want to drop me a line at stephrifai@docker.com we can find a time!

[nemchik]

Jumping in here, I do agree with @jpic that currently it doesn't seem like there's complete documentation regarding migrating from v1 to v2.

Just to fabricate an example of what I think would be useful:

docker-compose up -d has been replaced with docker compose up --background

I know -d is the same in v1 and v2, but what I'm getting at is anytime a command can't simply remove the - in docker-compose ... and expect functionally to be the same, documentation would be appreciated. Specifically I'd come looking for "what do I need to replace in order to get my project working again?" And even as a developer, in a situation like this (upgrading involving any kind of migration) my first instinct is going to be too look for a migration doc or hope for a well documented migration process in the change log, not to dig through commit history or source code.

I even looked at the compose switch repository and hoped to see exactly what it's doing aside from being an alias from docker-compose to docker compose (since it's described as converting things for you) but I can't really tell just by looking at the readme, which should be as far as a general user should be expected to go to find that information or an obvious link to where it's fully documented.

Thanks for reading. I'm happy to see compose v2 coming along nicely. I won't be making any efforts to switch from v1 to v2 with any of my production projects until the above is addressed, which hopefully happens before v1 is considered fully EOL.

P.s. I would love if compose we're just bundled with the cli, or could be installed via docker plugin install compose, but I can open a separate issue to discuss that if it's more appropriate.

[masi]

@jpic

the expansion of volumes just seems completely useless.

I thought that the format of the yml is the same for the new "docker compose" and the old "docker-compose".

Do you mean that the shorthand syntax has been dropped for "docker compose"?

Looking at https://docs.docker.com/compose/compose-file/compose-file-v3 and https://docs.docker.com/compose/cli-command/ one would think that at least the config files doe not need any changes.

[masi]

Ideally for devops, devs and sysops the only difference between "docker compose" and "docker-compose" is the hyphen.

Any arguments that become new (better?) names should be accepted in the old format with a deprecation notice printed in the shell. Is it so hard to accept old -d as an alternative to --background for one or two years?

But as others have argued here as there is no list of changes readily available it's hard to tell how much work users all around the world have to fix all their scripts.

PS: in the future I would expect that "docker compose" comes as part of the CLI as any other of the "docker" commands.

PPS: other than that I'm happy to see a Python helper script go.

[markus-ksg]

Breaking changes like the new spec for .env files ( https://github.com/docker/compose/issues/8607 ) are a no-go. In that case there is not even a way to have a .env file that a) contains a $ in a value, and b) is compatible with both V1 and V2 I.e. the only way to migrate to V2 is to make sure all team members and all docker services switch to V2 at once.

[nemchik]

In that case there is not even a way to have a .env file that ... b) is compatible with both V1 and V2

This is not 100% correct. I commented a workaround here https://github.com/docker/compose/issues/8763#issuecomment-959460133 this will allow you to use $ but it won't be evaluated.

The changes to how environment variables are handled in v2 are resulting in a pretty significant amount of seemingly duplicate issues being opened and there's very little triage happening to consolidate things so people can find relevant discussions about the topic.

I hope to see the environment file issues addressed before v2 is considered GA and v1 is fully considered EOL.

[scyto]

my 2 cents

I installed all my docker hosts using the official get-docker.sh script - my expectation (hope?) is that re-running that script (which is a supported version upgrade path scenario today) or running apt-get upgrade on systems that had used this would do the right thing and move from compose v1 to v2 - this assumes that v2 had broad functional parity with v1 in complex deployment - for example using swarm nodes, plugins etc

watching the last 6 mo of bugs fixed the pace of change is fantastic, but it doesn't seem ready to declare GA as i see plenty of feature parity breaking changes in the issues still marked as open.

[ndeloof]

@scyto docker-compose does not support swarm, only docker stack does and this is a distinct product. Compose v2 is available with recent release of our docker system packages for Linux.

[nemchik]

Compose v2 is available with recent release of our docker system packages for Linux.

Wait, is compose v2 built into docker cli already? So on a clean installed OS if someone used the get.docker.com script they would also have compose v2 installed?

[ndeloof]

yes, docker compose verb will be available to Linux installation by docker packages (actually, for the 20.10.x release this is a recommended sub-package, but should be installed for most users)

[scyto]

I am confused, I am running debian, installed docker via the get-docker.sh a couple of months ago have re-run script, and done apt-get update / upgrade i don't seem to have docker compose

alex@docker03:~$ sudo docker version
Client: Docker Engine - Community
 Version:           20.10.13
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 10 14:08:15 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.13
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       906f57f
  Built:            Thu Mar 10 14:06:05 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.10
  GitCommit:        2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[ndeloof]

@scyto docker info to list client plugins, should include "Docker Compose" Then, docker compose version

[scyto]

@scyto docker info to list client plugins, should include "Docker Compose"

Then, docker compose version

Definitely not installed. Only app buildx and scan.

[ndeloof]

oh indeed, the get-docker.sh script is not yet updated to also install compose. Need to make it explicit with apt-get install docker-compose-plugin

[markkrj]

@ndeloof do you have an ETA for when completions should be ready?

[ndeloof]

Completion for docker compose ... require a redesign of the completion in the Docker CLI. This is currently prototyped under https://github.com/docker/cli/pull/3429 but I can't give an estimate. If you use the "classic" docker-compose command (which compose v2 supports, by direct execution) you can use the v1 completion script.

[scyto]

oh indeed, the get-docker.sh script is not yet updated to also install compose. Need to make it explicit with apt-get install docker-compose-plugin

worked perfectly, I am now highly amused by all the threads 'on the web' / reddit / etc that are a multi step process based on this page.. https://docs.docker.com/compose/cli-command/ - maybe just adding an 'if you used get-docker.sh then...` would be worthwile?

anyhoo, this is awesome, i will update my own docs to reflect this :-)

[thaJeztah]

anyhoo, this is awesome, i will update my own docs to reflect this :-)

Glad it worked well!

We indeed have some docs to update for this (and the get.docker.com install script, as mentioned); as this went out with a patch release of docker 20.10, we tried to make it not too disruptive (e.g., don't make it a "required" dependency, so that users can "opt-in" to installing the package); the installation script at get.docker.com is a slightly more "opinionated'' way to install docker, and for that, we plan to make the installation automatic.

[rfay]

IMO it should be a required dependency :)

[scyto]

IMO it should be a required dependency :)

oh agree totally, was just thinking that was a higher mountain for me to climb, lol :-)

[ndeloof]

Last release was a "patch" release so we can't make such significant change as a "new required dependency". 22.x release will

[nemchik]

Where does this new package install the compose binary?

I ask because the readme indicates it could be in any of:

~/.docker/cli-plugins
/usr/local/lib/docker/cli-plugins
/usr/local/libexec/docker/cli-plugins
/usr/lib/docker/cli-plugins
/usr/libexec/docker/cli-plugins