An updated multi-arch image is pulled with wrong architecture

[TrueXakeP]

Describe the bug As I described in https://github.com/containrrr/watchtower/issues/1287#issuecomment-1159814148 Watchtower fetches the wrong architecture of multi-arch images than the container is configured.

To Reproduce docker-compose.yml:

version: "3.4"
services:
  httpd-base:
    image: httpd:2.4
    platform: linux/i386
    restart: "no"
    entrypoint:
      - sh
      - -c
      - exit
    profile:
      - pull-only
    labels:
      com.centurylinklabs.watchtower.scope: pull

docker pull httpd:2.4.53
docker image tag httpd:2.4.53 httpd:2.4
docker compose --profile pull-only up --no-start --no-deps --force-recreate httpd-base
# [+] Running 1/0
# ⠿ Container 42ffb5434485_42ffb5434485_1c-in-docker-httpd-base-1  Recreated

docker image inspect httpd:2.4 -f "{{.Architecture}}"
# 386

Container was recreated after previous experiments.

Run the update:

docker run --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /etc/localtime:/etc/localtime:ro \
-v /etc/timezone:/etc/timezone:ro \
containrrr/watchtower \
--scope pull \
--include-stopped \
--debug \
--run-once \
1c-in-docker-httpd-base-1

And this is an error:

docker image inspect httpd:2.4 -f "{{.Architecture}}"
# amd64

docker image inspect httpd:2.4.53 -f "{{.Architecture}}"
# 386

docker compose up --no-start --no-deps httpd-base
# [+] Running 0/0
#  ⠋ Container 1c-in-docker-httpd-base-1  Recreate                                                                                                                                                                                           0.0s
# Error response from daemon: image with reference httpd:2.4 was found but does not match the specified platform: wanted linux/386, actual: linux/amd64

Then my containers and images look like this:

```shell docker ps -a ``` ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES af4daecab294 httpd:2.4 "sh -c exit" 6 minutes ago Created 1c-in-docker-httpd-base-1 49a6db9bef82 containrrr/watchtower "/watchtower --scope…" 2 hours ago Exited (1) 26 minutes ago 1c-in-docker-watchtower-int-1 2f9cf991b06d containrrr/watchtower "/watchtower --scope…" 2 hours ago Exited (1) 26 minutes ago 1c-in-docker-watchtower-1 fc78613e637b nginx:1.20-alpine "sh -c exit" 2 hours ago Created 1c-in-docker-nginx-base-1 7c6a4473c0ef alpine:latest "sh -c exit" 2 hours ago Created 1c-in-docker-openvpn-base-1 e416b619ce4f alpine/socat "sh -c exit" 2 hours ago Created 1c-in-docker-dns-proxy-base-1 26b465323339 ubuntu:focal "sh -c exit" 2 hours ago Created 1c-in-docker-postgres-base-1 f53b84830da6 alpine/socat "socat UDP-LISTEN:53…" 2 hours ago Up 2 hours 1c-in-docker-dns-proxy-udp-1 63b57c25f3dc alpine/socat "socat TCP-LISTEN:53…" 2 hours ago Up 2 hours 1c-in-docker-dns-proxy-tcp-1 4a1993b915c7 docker "docker-entrypoint.s…" 2 hours ago Up 2 hours 1c-in-docker-cleanup-builder-1 61465f925ad1 docker "docker-entrypoint.s…" 2 hours ago Exited (143) 26 minutes ago 1c-in-docker-builder-1 e72fc2f1fb1f crazymax/fail2ban:latest "/entrypoint.sh fail…" 2 hours ago Up 2 hours (healthy) fail2ban 79343488bf1b kylemanna/openvpn "ovpn_run" 2 hours ago Up 2 hours 0.0.0.0:1555->1194/udp, :::1555->1194/udp openvpn 0a3194a32f4d apt-mirror "/bin/sh -c 'trap ex…" 2 hours ago Up 2 hours 1c-in-docker-pg-mirror-1 36d999aab0fa 1c-server:8.3.15-1700 "/entrypoint.sh /bin…" 47 hours ago Up 47 hours 80/tcp, 1540-1541/tcp, 1545/tcp, 1560-1591/tcp 1c-in-docker-1c-ras-1 250172c3bd07 1c-server:8.3.15-1700 "/entrypoint.sh rage…" 47 hours ago Up 47 hours 80/tcp, 1540-1541/tcp, 1545/tcp, 1560-1591/tcp 1c-server 21557b7c0152 nginx:1.20-alpine "/docker-entrypoint.…" 6 days ago Up 4 days 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp 1c-in-docker-nginx-1 c306058410ef certbot/certbot "/bin/sh -c 'trap ex…" 8 days ago Up 4 days 80/tcp, 443/tcp 1c-in-docker-certbot-1 6fa70d5f5ea4 postgrespro:13 "/entrypoint.sh post…" 8 days ago Up 4 days 5432/tcp 1c-postgrespro ``` ```shell docker image ls -a ``` ``` REPOSITORY TAG IMAGE ID CREATED SIZE 1c-server 8.3.15-1700 9378a4c273b4 6 days ago 2.05GB httpd 2.4 b260a49eebf9 7 days ago 145MB postgrespro 13 2d3076a0a6d7 8 days ago 452MB apt-mirror latest 50ab5bfd075d 8 days ago 162MB certbot/certbot latest dce0bfb74107 13 days ago 115MB docker latest 0208b15ab970 2 weeks ago 294MB ubuntu focal 20fffa419e3a 2 weeks ago 72.8MB httpd 2.4.53 6207b07928e9 3 weeks ago 141MB alpine latest e66264b98777 4 weeks ago 5.53MB nginx 1.20-alpine 6cf0c840a5a4 4 weeks ago 23.2MB alpine/socat latest aadd62c3c1fe 2 months ago 7.04MB containrrr/watchtower latest 333de6ea525a 5 months ago 16.9MB crazymax/fail2ban latest 6efb19d309d4 5 months ago 75.5MB kylemanna/openvpn latest 3d5a3627a7f6 18 months ago 15.3MB ``` ```shell docker image inspect b260a49eebf9 -f "{{.Architecture}}" # amd64 ```

Expected behavior The architecture of the httpd:2.4 (which is now 2.4.54) and httpd:2.4.53 images should match after the upgrade, and docker compose up --no-start --force-recreate httpd-base should correctly recreate the service.

Screenshots

Environment

uname -a
# Linux ubuntu-2gb-hel1-2 5.4.0-120-generic #136-Ubuntu SMP Fri Jun 10 13:40:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux`

lsb_release -a
# No LSB modules are available.
# Distributor ID: Ubuntu
# Description:    Ubuntu 20.04.4 LTS
# Release:        20.04
# Codename:       focal

docker -v
# Docker version 20.10.16, build aa7e414

docker compose version
# Docker Compose version v2.5.0

pwd
# /home/txp/1c-in-docker
# so docker compose project is '1c-in-docker'

Logs from running watchtower with the --debug option

``` time="2022-06-21T21:44:43+03:00" level=debug msg=pull time="2022-06-21T21:44:43+03:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized." time="2022-06-21T21:44:44+03:00" level=debug msg="Making sure everything is sane before starting" time="2022-06-21T21:44:44+03:00" level=info msg="Watchtower 1.4.0" time="2022-06-21T21:44:44+03:00" level=info msg="Using no notifications" time="2022-06-21T21:44:44+03:00" level=info msg="Only checking containers with name \"1c-in-docker-httpd-base-1\", in scope \"pull\"" time="2022-06-21T21:44:44+03:00" level=info msg="Running a one time update." time="2022-06-21T21:44:44+03:00" level=debug msg="Checking containers for updated images" time="2022-06-21T21:44:44+03:00" level=debug msg="Retrieving running, stopped and exited containers" time="2022-06-21T21:44:44+03:00" level=debug msg="Trying to load authentication credentials." container=/1c-in-docker-httpd-base-1 image="httpd:2.4" time="2022-06-21T21:44:44+03:00" level=debug msg="No credentials for httpd:2.4 found" config_file=/config.json time="2022-06-21T21:44:44+03:00" level=debug msg="Got image name: httpd:2.4" time="2022-06-21T21:44:44+03:00" level=debug msg="Checking if pull is needed" container=/1c-in-docker-httpd-base-1 image="httpd:2.4" time="2022-06-21T21:44:44+03:00" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/" time="2022-06-21T21:44:44+03:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized" time="2022-06-21T21:44:44+03:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io time="2022-06-21T21:44:44+03:00" level=debug msg="Setting scope for auth token" image=httpd scope="repository:library/httpd:pull" time="2022-06-21T21:44:44+03:00" level=debug msg="No credentials found." time="2022-06-21T21:44:45+03:00" level=debug msg="Parsing image ref" host=index.docker.io image=httpd normalized="docker.io/library/httpd:2.4" tag=2.4 time="2022-06-21T21:44:45+03:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/library/httpd/manifests/2.4" time="2022-06-21T21:44:45+03:00" level=debug msg="Found a remote digest to compare with" remote="sha256:f899e432292e4ee92772d35e43b2e3dcf30042b1c6385d33f00a9300c69ee729" time="2022-06-21T21:44:45+03:00" level=debug msg=Comparing local="sha256:c479bec894c5a7f8878b28e52d03cc95b1e784612ecd01ac7c7394fc5fa2e6e2" remote="sha256:f899e432292e4ee92772d35e43b2e3dcf30042b1c6385d33f00a9300c69ee729" time="2022-06-21T21:44:45+03:00" level=debug msg="Digests did not match, doing a pull." time="2022-06-21T21:44:45+03:00" level=debug msg="Pulling image" container=/1c-in-docker-httpd-base-1 image="httpd:2.4" time="2022-06-21T21:44:54+03:00" level=info msg="Found new httpd:2.4 image (b260a49eebf9)" time="2022-06-21T21:44:54+03:00" level=info msg="Session done" Failed=0 Scanned=1 Updated=1 notify=no time="2022-06-21T21:44:54+03:00" level=info msg="Waiting for the notification goroutine to finish" notify=no ```

Additional context Platform of a container is always "linux"

docker container inspect f53b84830da6 -f "{{.Platform}}"
# linux

[github-actions[bot]]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[stoinov]

I am experiencing similar issue - due to being stuck with older version of docker (19.03.15), manifest are not properly parsed Typical output:

no matching manifest for linux/arm/v8 in the manifest list entries

I have to manually set --platform arm64 on my deployed containers.

Watchtower does not seem to honor that and just relies on the docker mechanism to handle platforms, so it fails to pull any new images as the pull does not return anything.

Can you make it use the `--platform parameter if possible? Or have a general ENV for setting this globally?

[piksel]

Can you make it use the `--platform parameter if possible? Or have a general ENV for setting this globally?

That seems a bit less useful, as you might only have a single container that you need to use another platform for. But if the container was created with an explicitly defined platform, watchtower should be able to retain that platform. The only issue I see, is correctly identifying that it's using a non-default platform.
I'm currently thinking it could be done by:

1. Determine the docker host platform, should be possible with some API call
2. Check the current image for multi-arch to determine if the default platform was available, if not, it was explicitly set.

I don't know if 2. is possible though, and without it we would pin a container to the previous platform, even though a native one becomes available. But perhaps that's not that much of an issue, especially if we put that behaviour behind a flag, like --retain-platform.

[stoinov]

My suggestion - check the container architecture from an inspect and use it as platform. This can be a global flag or per container tag.

[simskij]

My suggestion - check the container architecture from an inspect and use it as platform. This can be a global flag or per container tag.

None of that should be needed. The image should have a platform attached to it, which should be the one it pulls next time. I strongly feel there is something else at play here as we rely on the docker engine and dockerhub to do the heavy lifting.

[piksel]

@simskij well, the image does have a platform, but we are not adding that to the pull API call. It should just be a matter of adding this to dockerClient.PullImage:

    if persistPlatform {
        opts.Platform = container.ContainerInfo().Platform
    }