[BUG] Docker compose watch builds multiple times the sources

asiragusa commented 1 year ago

Description

Docker compose watch builds multiple time the sources even if two or more services share the same build configuration. In my case I have 10+ services that use the same Dockerfile in a mono repo and this slows down significantly the build process.

Moreover if one of the rebuild targets is changed while docker compose watch is not running, the build is not triggered at the start.

Steps To Reproduce

A simplified version of my docker-compose.yaml below:

version: '2.22'
services:
  service_a:  
    build:
      context: .
      target: build

    develop:
      watch:
        - action: sync
          path: .
          target: /app
          ignore:
            - node_modules/
        - action: rebuild
          path: package.json
        - action: rebuild
          path: yarn.lock

    command:
      - yarn
      - run
      - serviceA

  service_b:  
    build:
      context: .
      target: build

    develop:
      watch:
        - action: sync
          path: .
          target: /app
          ignore:
            - node_modules/
        - action: rebuild
          path: package.json
        - action: rebuild
          path: yarn.lock

    command:
      - yarn
      - run
      - serviceB

Compose Version

Docker Compose version v2.22.0-desktop.2

Docker Environment

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     ~/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.22.0-desktop.2
    Path:     ~/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     ~/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     ~/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.8
    Path:     ~/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     ~/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     ~/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.7
    Path:     ~/.docker/cli-plugins/docker-scout

Server:
 Containers: 29
  Running: 19
  Paused: 0
  Stopped: 10
 Images: 24
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 17
 Total Memory: 7.67GiB
 Name: docker-desktop
 ID: e8b2ead2-8e94-4cf7-8d4c-37ce75b3f3e7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

No response

github-actions[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

sfc-gh-kbregula commented 2 months ago

As a workaround you can create a new service that will be responsible for building the image, and other services will use this image by name.

services:
  my-app-image:
    build:
      context: .
      dockerfile: Dockerfile
    image: ${COMPOSE_PROJECT_NAME}-my-app
    command: ['echo', 'The docker image (${COMPOSE_PROJECT_NAME}-docparser) is ready to be used.']
    develop:
      watch:
        - path: ./requirements.txt
          action: rebuild

  my-app-instance-1:
    depends_on:
      my-app-image:
        condition: service_completed_successfully
        required: false
    environment:
      INSTANCE_NAME: instance1
    build:
      context: .
      dockerfile: Dockerfile
    image: ${COMPOSE_PROJECT_NAME}-my-app
    ports:
      - "3001:3001"
    command: [ 'flask', 'run', '--host=0.0.0.0', '--port=3001' ]

  my-app-instance-2:
    depends_on:
      my-app-image:
        condition: service_completed_successfully
        required: false

    image: ${COMPOSE_PROJECT_NAME}-my-app
    environment:
      INSTANCE_NAME: instance2
    ports:
      - "3002:3002"
    command: [ 'flask', 'run', '--host=0.0.0.0', '--port=3002' ]

Inspiration: https://stackoverflow.com/questions/40899236/how-to-prevent-docker-compose-building-the-same-image-multiple-times

But after that you can't use sync action in services until the issue is resolved: https://github.com/docker/compose/issues/12065

stale[bot] commented 2 months ago

This issue has been automatically marked as not stale anymore due to the recent activity.

jaredjj3 commented 2 months ago

This was happening on containers that are supposed to exit. A workaround that worked for me for those containers was to prevent the container from exiting.

command: ["sh", "-c", "bin/generate && tail -f /dev/null"]

However, when sending a SIGINT signal to kill the docker process, these containers no longer shutdown gracefully — they time out shutdown after 10 seconds.

docker / compose