[BUG] --force-recreate does not use the newly built image

ZEB1CLJ commented 3 months ago

Description

docker compose recreates container with old (cached?) version of image, although I run it with both --force-recreate and --build arguments.

I've noticed that the following changes to the Dockerfile have absolutely no effect on the recreated containers (i.e. the containers seem not to pick up the changes)>

changes in files used by the COPY instruction
changes to environment variables exported using the ENV directive

As others have previously stated (here: https://github.com/docker/compose/issues/4273), I have to manually remove the containers if I want docker compose to recreate them properly. (By the way: what exactly does recreate do? Does it delete the container or not?)

Steps To Reproduce

The contents of the files that I use are fairly simple, so they are listed below:

Save all these files in a folder, then call the runscript.sh script. Image build should start and a container should be started too.
Go inside the jenkins-autoconfigured container, echo an environment variable (e.g.: JAVA_OPTS)
Change JAVA_OPTS (or the env var that you printed at step 2) to something else
Call runscript.sh again. This will rebuild the image and "recreate" the container.
docker exec inside the recreated jenkins-autoconfigured and notice that the env var JAVA_OPTS is the same, even if you have rebuilt the image using docker compose.
For completeness, docker run -ti the rebuilt docker image and notice how the JAVA_OPTS variable is the correct (modified) one.

Dockerfile:

FROM jenkins/jenkins:latest

ARG JENKINS_ADMIN_ID
ARG JENKINS_ADMIN_PASSWORD

ENV JAVA_OPTS="${JAVA_OPTS} -Djenkins.install.runSetupWizard=false"
# Jenkins configuration file
ENV CASC_JENKINS_CONFIG /var/jenkins_home/casc.yaml
ENV JENKINS_ADMIN_ID=${JENKINS_ADMIN_ID}
ENV JENKINS_ADMIN_PASSWORD=${JENKINS_ADMIN_PASSWORD}

# Install jenkins plugins from "plugins.txt"
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli \
        -f /usr/share/jenkins/ref/plugins.txt \
        --verbose

COPY casc.yaml /var/jenkins_home/casc.yaml

plugins.txt:

ant:latest
antisamy-markup-formatter:latest
build-timeout:latest
cloudbees-folder:latest
configuration-as-code:latest
credentials-binding:latest
email-ext:latest
git:latest
github-branch-source:latest
gradle:latest
ldap:latest
mailer:latest
matrix-auth:latest
pam-auth:latest
pipeline-github-lib:latest
pipeline-stage-view:latest
ssh-slaves:latest
timestamper:latest
workflow-aggregator:latest
ws-cleanup:latest

casc.yaml:

jenkins:
  securityRealm:
    local:
      allowsSignup: false
      users:
        - id: fsdfasd
          password: fsadfas

Dockercompose:

services:
  jenkins-autoconfigured:
    image: jenkins:jcasc
    network_mode: host
    container_name: jenkins-autoconfigured
    build:
      dockerfile: ./Dockerfile
      context: .
      network: host
      args:
        - JENKINS_ADMIN_ID=admin
        - JENKINS_ADMIN_PASSWORD=admin

runscript.sh:

docker compose \
    --file Dockercompose.yaml \
    --progress plain \
    up \
    --detach \
    --build \
    --force-recreate

Compose Version

Docker Compose version v2.28.1

Docker Environment

Client: Docker Engine - Community
 Version:    27.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 16
  Running: 16
  Paused: 0
  Stopped: 0
 Images: 48
 Server Version: 27.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: /etc/docker/seccomp.json
 Kernel Version: 5.15.0-101-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.21GiB
 Name: THIS INFO HAS BEEN HIDDEN BY THE REPORTER - CONFIDENTIAL
 ID: e3331e38-c563-46d8-8a80-b55ee4121371
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http://127.0.0.1:3128/
 HTTPS Proxy: http://127.0.0.1:3128/
 No Proxy: THIS INFO HAS BEEN HIDDEN BY THE REPORTER - CONFIDENTIAL
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Anything else?

No response

jhrotko commented 3 months ago

Hello @ZEB1CLJ ,

I followed the steps as you suggested: I changed $JAVA_OPTS in Dockerfile to something else and after running the runscript.sh when running exec in the container the $JAVA_OPTS is updated as expected. Did I miss something?

ZEB1CLJ commented 3 months ago

Hello! Indeed, the thing works as you mentioned. However, I retested the example and saw that there actually is a problem with files not being updated after a COPY instruction is re-executed during build. So the env vars were properly updated, but the files were not.

But I found out that the Jenkins Docker image defines an anonymous volume that gets mounted automatically when using docker compose up. It turns out that docker compose up preserves anonymous volumes when recreating containers, so that was why the file didn't change.

I finally figured out that using --renew-anon-volumes does the trick.

docker / compose