[BUG] Compose v2 is recreating ALL containers where there is a change to one of the "depends_on" in the hierarchy (bring back compose v1 behaviour)

[captainfalcon23]

Description

Separating out an issue described on https://github.com/docker/compose/issues/9600

Basically, this comment (https://github.com/docker/compose/issues/9600#issuecomment-1403124191) encompasses this issue. Sometimes containers are completely recreated when we do not expect them to, and when using "depends_on", and a dependant container changes in some way, all of the children are restarted. In compose v1.X, this is NOT the case, and ONLY the changed container is recreated and all others are untouched.

Either this is a bug, or a feature request to bring back compose v1 behaviour (at least with a flag).

Steps To Reproduce

Refer https://github.com/docker/compose/issues/9600#issuecomment-1403124191

Compose Version

Docker Compose version v2.16.0

Docker Environment

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.16.0)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 4
  Running: 4
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.80.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.06GiB
 Name: XXX
 ID: GR4X:Q35F:5E4H:TGFY:O34K:JEYN:WYBW:3QY5:4G7N:TXP6:4GLE:QOG2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

Refer to https://github.com/docker/compose/issues/9600 for alot of history.

[ndeloof]

With https://github.com/docker/compose/pull/10284 you can now explicitly opt-in/out for dependent container to be restarted when a service is recreated.

[captainfalcon23]

Can you please share how to configure this in compose file? Or link to the docs :) I couldn't see a reference in the MR.

[ndeloof]

see https://github.com/compose-spec/compose-spec/blob/master/spec.md#long-syntax-1

[captainfalcon23]

Hi @ndeloof , I finally got some time to test this today and it didn't work.

These are the steps I followed:

# compose ver:
Docker Compose version v2.17.0-rc.1

docker compose down
docker system prune --all
# clean slate for testing

# update docker-compose.yml as per example below
docker compose up -d
# pulls all 4 images and creates all 4 containers as expected

# Run "docker compose up -d" multiple times, no containers recreated needlessly - good!
docker compose up -d
docker compose up -d
docker compose up -d

# Then, I changed the image tag of service-4 in docker-compose.yml
docker compose up -d
# ALL the dependent containers got recreated :( 
[+] Running 4/4
 ✔ Container dro-dns-fe-pdns-publisher-1  Running                                                                                           0.0s
 ✔ Container dro-dns-fe-syslog-ng-1       Started                                                                                          21.8s
 ✔ Container dro-dns-fe-pdns-recursor-1   Started                                                                                          21.2s
 ✔ Container dro-dns-fe-dnsdist-1         Started                                                                                          10.8s

# Run "docker compose up -d" multiple times, no containers recreated needlessly - good!
docker compose up -d
docker compose up -d
docker compose up -d
docker compose up -d

# Again, modified the image tag of service-4, ALL the dependent containers got recreated :(

# Run "docker compose up -d" multiple times, no containers recreated needlessly - good!
docker compose up -d
docker compose up -d
docker compose up -d
docker compose up -d

Sample docker-compose.yml file:

version: '3.7'
services:
 service-1:
  image: "XXXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/image1:1.7.0-pdns-1026"
  restart: always
  env_file:
    - .env_X
  network_mode: host
  depends_on:
   service-2:
     condition: service_started
     restart: false
   service-3:
     condition: service_started
     restart: false
 service-2:
  image: "XXXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/image2:4.5.10-pdns-1023"
  restart: always
  env_file:
    - .env_X
  network_mode: host
  volumes:
    - /mnt/data/service-4/dev:/service-4
    - /mnt/data/service-2:/var/service-2
  depends_on:
    service-4:
      condition: service_started
      restart: false
    service-3:
      condition: service_started
      restart: false
 service-3:
  image: "XXXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/image3:1.1.0-15-pdns-1024"
  restart: always
  env_file:
    - .env_X
  network_mode: host
  volumes:
    - /mnt/data/a:/a
 service-4:
  image: "XXXXXX.dkr.ecr.ap-southeast-2.amazonaws.com/image4:3.24-pdns-1022"
  restart: always
  env_file:
    - .env_X
  network_mode: host
  volumes:
    - /mnt/data/a/dev:/a
    - /mnt/data/b:/b
    - /mnt/data/c:/c

Have I made a mistake in the config, or is it just not working as expected?

[andryushchenko]

Having same issue as decribed in https://github.com/docker/compose/issues/10308#issuecomment-1461332080 with Compose v2.24.6 (latest from Ubuntu 24.04 feed) Is this a new by design bahaviour or how to avoid this?

[ndeloof]

@andryushchenko see https://github.com/docker/compose/issues/10308#issuecomment-1447781552

[andryushchenko]

@ndeloof Please can you provide the example of correct usage this option? I have the same situation as @captainfalcon23 - restart:false is set but conainers still being restarted

[koniiiik]

I have now spent some time looking into this issue, and as far as I can tell, the restart option of depends_on only applies to the docker compose restart command. At least, that's the only place that the implementation of #10284 touched. However, that does not come into effect when docker compose applies a new convergence on the entire dependency tree.

There's a pair of flags for docker compose up that are supposed to affect whether dependencies are recreated: --always-recreate-deps and --no-recreate; without the presence of either of these two, the recreation strategy should default to RecreateDiverged defined here, as selected by createOptions.dependenciesRecreateStrategy.

Unfortunately, the observed behavior seems to be as if --always-recreate-deps was on by default, but I'm not familiar enough with the internals of the convergence implementation to figure out why RecreateDiverged does not appear to apply.