search

containrrr / watchtower

A process for automating Docker container base image updates.
https://containrrr.dev/watchtower/
Apache License 2.0
19.05k stars 847 forks source link

Healtcheck not updated #1601

Closed wehrstedt closed 11 months ago

wehrstedt commented 1 year ago

Describe the bug

An image specifies the health check with various arguments, e. g. the following health check:

HEALTHCHECK --interval=5s --timeout=2s --start-period=10s --retries=3 CMD [ "/bin/bash", "/healthcheck.sh" ]

container: ($> docker inspect kjn2kj3498)

// ...
"Healthcheck": {
  "Test": [
    "CMD",
    "/bin/bash",
    "/healthcheck.sh"
  ],
  "Interval": 5000000000,
  "Timeout": 2000000000,
  "StartPeriod": 10000000000,
  "Retries": 3
},

The image author updates the health check parameters, e. g. timeout will be increased to 15s, and retries to 4:

HEALTHCHECK --interval=5s --timeout=15s --start-period=10s --retries=4 CMD [ "/bin/bash", "/healthcheck.sh" ]

Watchtower updates the container by pulling the new image and recreate a new container of the new image. But the new container runs with the old healthcheck arguments instead of the updated ones:

$> docker inspect ui43j3na

// ...
"Healthcheck": {
  "Test": [
    "CMD",
    "/bin/bash",
    "/healthcheck.sh"
  ],
  "Interval": 5000000000,
  "Timeout": 2000000000,
  "StartPeriod": 10000000000,
  "Retries": 3
},

If you manually re-create the container, e. g. using docker-compose pull and docker-compose up -d, the updated health check parameters will be applied to the new container:

$> docker inspect asdn2904ds

// ...
  "Healthcheck": {
    "Test": [
        "CMD",
        "/bin/bash",
        "/healthcheck.sh"
    ],
    "Interval": 60000000000,
    "Timeout": 15000000000,
    "StartPeriod": 600000000000,
    "Retries": 4
  },

Steps to reproduce

  1. Create a simplified dockerfile: 
    
FROM ubuntu

HEALTHCHECK --interval=5s --timeout=2s --start-period=10s --retries=3 CMD [ "/usr/bin/sleep", "1s" ]

ENTRYPOINT [ "/usr/bin/sleep", "10m" ]

2. build the image: `docker build --no-cache -t healthcheck-image .`
3. publish the image: `docker push healthcheck-image`
4. Create a container + watchtower with the following compose file:
```yaml
version: '3'
services:
  healthcheck-test:
    image: healtcheck-image
    labels:
      - com.centurylinklabs.watchtower.enable=true

  watchtower:
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: --interval 10
    environment:
      - WATCHTOWER_LABEL_ENABLE=true
  1. Inspect the container to check the healthcheck configuration 
    $> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 2s 10s 3}
  2. Update the healthcheck timeout and retries Parameter 
    
FROM ubuntu

HEALTHCHECK --interval=5s --timeout=15s --start-period=10s --retries=4 CMD [ "/usr/bin/sleep", "1s" ]

ENTRYPOINT [ "/usr/bin/sleep", "10m" ]

7. rebuild and republish the image
8. Wait for watchtower to update the container
9. re-check the container healthcheck configuration
```sh
$> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 2s 10s 3}

Expected:

$> docker inspect --format "{{ .Config.Healthcheck }}" container-id
{[CMD /usr/bin/sleep 1s] 5s 15s 10s 4}

Do the same with docker-compose up -d --force-recreate. The expected healthcheck configuration will be applied.

Expected behavior

The container should be recreated with the updated healtcheck configuration

Screenshots

No response

### Environment ``` docker info Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.10.2 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.16.0 Path: /usr/libexec/docker/cli-plugins/docker-compose scan: Docker Scan (Docker Inc.) Version: v0.23.0 Path: /usr/libexec/docker/cli-plugins/docker-scan Server: Containers: 9 Running: 2 Paused: 0 Stopped: 7 Images: 347 Server Version: 23.0.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640 runc version: v1.1.4-0-g5fd4c4d init version: de40ad0 Security Options: seccomp Profile: builtin Kernel Version: 5.10.102.1-microsoft-standard-WSL2 Operating System: Ubuntu 20.04.6 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 12.27GiB Name: Wehrstedt ID: DDGC:2DRA:UQYD:5LEP:PAUW:AA4Y:XA6G:MJD5:SCWX:U6MM:3D5Z:Z2IV Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 20 Base: 192.168.0.0/16, Size: 24 WARNING: No blkio throttle.read_bps_device support WARNING: No blkio throttle.write_bps_device support WARNING: No blkio throttle.read_iops_device support WARNING: No blkio throttle.write_iops_device support ```

Your logs

Attaching to watchtower-test_watchtower_1, watchtower-test_healthcheck-test_1
watchtower_1        | time="2023-03-21T11:43:49Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
watchtower_1        | time="2023-03-21T11:43:50Z" level=debug msg="Making sure everything is sane before starting"
watchtower_1        | time="2023-03-21T11:43:50Z" level=debug msg="Retrieving running containers"
watchtower_1        | time="2023-03-21T11:43:50Z" level=debug msg="There are no additional watchtower containers"
watchtower_1        | time="2023-03-21T11:43:50Z" level=debug msg="Watchtower HTTP API skipped."
watchtower_1        | time="2023-03-21T11:43:50Z" level=info msg="Watchtower 1.5.3"
watchtower_1        | time="2023-03-21T11:43:50Z" level=info msg="Using no notifications"
watchtower_1        | time="2023-03-21T11:43:50Z" level=info msg="Only checking containers using enable label"
watchtower_1        | time="2023-03-21T11:43:50Z" level=info msg="Scheduling first run: 2023-03-21 11:44:00 +0000 UTC"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Checking containers for updated images"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Retrieving running containers"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Trying to load authentication credentials." container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Loaded auth credentials for user <user>, on registry <registry-url>/healtcheck-image:latest, from file /config.json"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Got image name: <registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Credentials loaded"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Checking if pull is needed" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Building challenge URL" URL="https://<registry-url>/v2/"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://<registry-url>/jwt/auth\",service=\"container_registry\"" status="401 Unauthorized"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Checking challenge header content" realm="https://<registry-url>/jwt/auth" service=container_registry
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Setting scope for auth token" image=<registry-url>/healtcheck-image scope="repository:<registry-url>/healtcheck-image:pull"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Credentials found."
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Parsing image ref" host=<registry-url> image=healtcheck-image normalized="<registry-url>/healtcheck-image:latest" tag=latest
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://<registry-url>/v2/healtcheck-image/manifests/latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Could not do a head request for \"<registry-url>/healtcheck-image:latest\", falling back to regular pull." container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://<registry-url>/jwt/auth\\\",service=\\\"container_registry\\\",scope=\\\"repository:healtcheck-image:pull\\\",error=\\\"insufficient_scope\\\"\"" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=debug msg="Pulling image" container=/watchtower-test_healthcheck-test_1 image="<registry-url>/healtcheck-image:latest"
watchtower_1        | time="2023-03-21T11:44:30Z" level=info msg="Found new <registry-url>/healtcheck-image:latest image (feb17fe278fe)"
watchtower_1        | time="2023-03-21T11:44:30Z" level=info msg="Stopping /watchtower-test_healthcheck-test_1 (f29ee537d965) with SIGTERM"
watchtower_1        | time="2023-03-21T11:44:40Z" level=debug msg="Skipped another update already running."
watchtower_1        | time="2023-03-21T11:44:40Z" level=debug msg="Scheduled next run: 2023-03-21 11:44:50 +0000 UTC"
watchtower_1        | time="2023-03-21T11:44:40Z" level=debug msg="Removing container f29ee537d965"
watchtower-test_healthcheck-test_1 exited with code 137
watchtower_1        | time="2023-03-21T11:44:41Z" level=info msg="Creating /watchtower-test_healthcheck-test_1"
watchtower_1        | time="2023-03-21T11:44:41Z" level=debug msg="Starting container /watchtower-test_healthcheck-test_1 (de887b1bc2fc)"
watchtower_1        | time="2023-03-21T11:44:41Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=1 notify=no

Additional context

No response

simskij commented 1 year ago

Interesting! Yeah, I think we copy the current container spec to the teeth, including any healthcheck declarations that it might have. Will have to look into how to honor changes to it.