search

containrrr / watchtower

A process for automating Docker container base image updates.
https://containrrr.dev/watchtower/
Apache License 2.0
19.71k stars 869 forks source link

Container does not get updated #1999

Open realies opened 4 months ago

realies commented 4 months ago

Describe the bug

When the image of a docker compose service is updated, it gets pulled, but the docker container does not get recreated or stopped. I'm not sure if it's related to compose.

Steps to reproduce

version: "3"
services:
  watchtower:
    container_name: watchtower
    image: containrrr/watchtower
    restart: unless-stopped
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_SCHEDULE=0 20 5 * * *
      - WATCHTOWER_REMOVE_VOLUMES=true
      - WATCHTOWER_DISABLE_CONTAINERS=wake-on-lan-service
      - WATCHTOWER_NOTIFICATIONS=shoutrrr
      - WATCHTOWER_NOTIFICATION_URL=x
      - WATCHTOWER_DEBUG=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Expected behavior

The container stops and gets recreated with the latest image

Screenshots

No response

Environment

Your logs

watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Parsing image ref" host=index.docker.io image=zoeyvid/npmplus normalized=docker.io/zoeyvid/npmplus tag=latest
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/zoeyvid/npmplus/manifests/latest"
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Found a remote digest to compare with" remote="sha256:661f67996f2293717f8f82dfc7b546494d63bea5689c30aa90aa1bd71b4abeb4"
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg=Comparing local="sha256:adb7e1f2d47724ceb27e494a1f2db3d21401b08311ccba2393b2b9e79017fdfa" remote="sha256:661f67996f2293717f8f82dfc7b546494d63bea5689c30aa90aa1bd71b4abeb4"
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Digests did not match, doing a pull."
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Pulling image" container=/npmplus image="zoeyvid/npmplus:latest"
watchtower  | time="2024-07-14T05:26:04Z" level=info msg="Found new zoeyvid/npmplus:latest image (43ff4b4c4210)"
watchtower  | time="2024-07-14T05:26:04Z" level=debug msg="Trying to load authentication credentials." container=/dpaste image="darrenofficial/dpaste:latest"
watchtower  | time="2024-07-14T05:26:04Z" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
watchtower  | time="2024-07-14T05:26:04Z" level=debug msg="Got image name: darrenofficial/dpaste:latest"
watchtower  | time="2024-07-14T05:26:04Z" level=debug msg="Checking if pull is needed" container=/dpaste image="darrenofficial/dpaste:latest"
...
watchtower  | time="2024-07-14T05:26:10Z" level=info msg="Session done" Failed=0 Scanned=7 Updated=1 notify=no

Additional context

No response

github-actions[bot] commented 4 months ago

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

ciberboy commented 3 months ago

I have a similar issue.

This is the configuration I am using: docker create \ --name watchtower \ -e PUID=0 \ -e PGID=0 \ -e WATCHTOWER_CLEANUP=TRUE \ -e WATCHTOWER_DEBUG=TRUE \ -e WATCHTOWER_POLL_INTERVAL=300 \ -v /var/run/docker.sock:/var/run/docker.sock \ -v $HOME/.docker/config.json:/config.json \ --restart unless-stopped \ containrrr/watchtower

These are the logs: time="2024-08-01T12:32:04Z" level=debug msg="Trying to load authentication credentials." container=/photoprism image="photoprism/photoprism:latest" time="2024-08-01T13:04:59Z" level=debug msg="Retrieving running containers" time="2024-08-01T12:32:04Z" level=debug msg="Loaded auth credentials for user XXXXXXXXXXXXX, on registry index.docker.io, from file /config.json" time="2024-08-01T12:32:04Z" level=debug msg="Got image name: photoprism/photoprism:latest" time="2024-08-01T12:32:04Z" level=debug msg="Credentials loaded" time="2024-08-01T12:32:04Z" level=debug msg="Checking if pull is needed" container=/photoprism image="photoprism/photoprism:latest" time="2024-08-01T12:32:04Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/" time="2024-08-01T12:32:05Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized" time="2024-08-01T12:32:05Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io time="2024-08-01T12:32:05Z" level=debug msg="Setting scope for auth token" image=docker.io/photoprism/photoprism scope="repository:photoprism/photoprism:pull" time="2024-08-01T12:32:05Z" level=debug msg="Credentials found." time="2024-08-01T12:32:05Z" level=debug msg="Parsing image ref" host=index.docker.io image=photoprism/photoprism normalized=docker.io/photoprism/photoprism tag=latest time="2024-08-01T12:32:05Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/photoprism/photoprism/manifests/latest" time="2024-08-01T12:32:06Z" level=debug msg="Found a remote digest to compare with" remote="sha256:b8dc65e14808021dff7fdb60723fe7e7cd224db1fd356c4e3f1892fa4912e4e4" time="2024-08-01T12:32:06Z" level=debug msg=Comparing local="sha256:3b6a64d86abb566b5314dc7b168476e421ca7322b9102c1bd9c79834c6bc6756" remote="sha256:b8dc65e14808021dff7fdb60723fe7e7cd224db1fd356c4e3f1892fa4912e4e4" time="2024-08-01T12:32:06Z" level=debug msg="Digests did not match, doing a pull." time="2024-08-01T12:32:06Z" level=debug msg="Pulling image" container=/photoprism image="photoprism/photoprism:latest" time="2024-08-01T12:32:07Z" level=debug msg="No new images found for /photoprism" time="2024-08-01T12:32:07Z" level=debug msg="Trying to load authentication credentials." container=/mariadb image="mariadb:10.11" time="2024-08-01T12:32:07Z" level=debug msg="Loaded auth credentials for user XXXXXXXXXXXXX, on registry index.docker.io, from file /config.json" time="2024-08-01T12:32:07Z" level=debug msg="Got image name: mariadb:10.11" time="2024-08-01T12:32:07Z" level=debug msg="Credentials loaded" time="2024-08-01T12:32:07Z" level=debug msg="Checking if pull is needed" container=/mariadb image="mariadb:10.11" time="2024-08-01T12:32:07Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/" time="2024-08-01T12:32:08Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized" time="2024-08-01T12:32:08Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io time="2024-08-01T12:32:08Z" level=debug msg="Setting scope for auth token" image=docker.io/library/mariadb scope="repository:library/mariadb:pull" time="2024-08-01T12:32:08Z" level=debug msg="Credentials found." time="2024-08-01T12:32:08Z" level=debug msg="Parsing image ref" host=index.docker.io image=library/mariadb normalized=docker.io/library/mariadb tag=10.11 time="2024-08-01T12:32:08Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/library/mariadb/manifests/10.11" time="2024-08-01T12:32:08Z" level=debug msg="Found a remote digest to compare with" remote="sha256:75f6e61397758489d1dccf95db33b6b49ebfc7ec1253d40060fdf8ceb7f938a3" time="2024-08-01T12:32:08Z" level=debug msg=Comparing local="sha256:385c0ba6270307511a511ad915350736602c33fafb3f63f663c853920424c365" remote="sha256:75f6e61397758489d1dccf95db33b6b49ebfc7ec1253d40060fdf8ceb7f938a3" time="2024-08-01T12:32:08Z" level=debug msg="Digests did not match, doing a pull." time="2024-08-01T12:32:08Z" level=debug msg="Pulling image" container=/mariadb image="mariadb:10.11" time="2024-08-01T12:32:11Z" level=debug msg="No new images found for /mariadb" time="2024-08-01T12:32:11Z" level=debug msg="This is the watchtower container /watchtower" time="2024-08-01T12:32:11Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no time="2024-08-01T12:32:11Z" level=debug msg="Scheduled next run: 2024-08-01 12:36:48 +0000 UTC" time="2024-08-01T12:33:20Z" level=info msg="Waiting for running update to be finished..." time="2024-08-01T13:04:58Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized." time="2024-08-01T13:04:59Z" level=debug msg="Making sure everything is sane before starting"

There's a specific message regarding authentication, but I made sure I am using a current and valid docker token:

    "auths": {
            "https://index.docker.io/v1/": {
                    "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX>
            },
            "https://index.docker.io/v2/": {}
    }

} (I have replaced the token generated by the login action with X)

I am using for all my docker images the platform tag for x64. I don't know if that is that is the root cause of the issue, but I doubt it.

Additionally, watchtower was working fine up until a month ago. I haven't changed nothing in the meantime.

Is there a chance someone can help?

Thank you in advance for your help.

ciberboy commented 3 months ago

Anyone? Please help!

PopC0rnwalk commented 3 months ago

Since las 2-3 month I think, container images are not getting updated.

Checking the logs I see that watchtower is detecting same image hash, somehow it is not getting the new image hash from index.docker.io.

I have been checking the images in the docker hub and they have been updated, watchtowerr is not getting that info and new hash.

Here are some logs:

time="2024-08-11T05:30:01+02:00" level=debug msg="Trying to load authentication credentials." container=/website-wordpress-1 image="wordpress:latest" time="2024-08-11T05:30:01+02:00" level=debug msg="No credentials for index.docker.io found" config_file=/config.json time="2024-08-11T05:30:01+02:00" level=debug msg="Got image name: wordpress:latest" time="2024-08-11T05:30:01+02:00" level=debug msg="Checking if pull is needed" container=/website-wordpress-1 image="wordpress:latest" time="2024-08-11T05:30:01+02:00" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/" time="2024-08-11T05:30:01+02:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized" time="2024-08-11T05:30:01+02:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io time="2024-08-11T05:30:01+02:00" level=debug msg="Setting scope for auth token" image=docker.io/library/wordpress scope="repository:library/wordpress:pull" time="2024-08-11T05:30:01+02:00" level=debug msg="No credentials found." time="2024-08-11T05:30:01+02:00" level=debug msg="Parsing image ref" host=index.docker.io image=library/wordpress normalized=docker.io/library/wordpress tag=latest time="2024-08-11T05:30:01+02:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/library/wordpress/manifests/latest" time="2024-08-11T05:30:02+02:00" level=debug msg="Found a remote digest to compare with" remote="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb627d46634" time="2024-08-11T05:30:02+02:00" level=debug msg=Comparing local="sha256:4bc1649284319b09f6f8da1a181616c196f4e225ad9a7d7901b179e3fb421070" remote="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb62 7d46634" time="2024-08-11T05:30:02+02:00" level=debug msg=Comparing local="sha256:6c49d06a62950e6866c0fc9ee0fed6b0605da895db3ec6fa1036660a054a5215" remote="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb62 7d46634" time="2024-08-11T05:30:02+02:00" level=debug msg=Comparing local="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb627d46634" remote="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb62 7d46634" time="2024-08-11T05:30:02+02:00" level=debug msg="Found a match" time="2024-08-11T05:30:02+02:00" level=debug msg="No pull needed. Skipping image." time="2024-08-11T05:30:02+02:00" level=debug msg="No new images found for /website-wordpress-1"

cattomgithub commented 3 months ago

Unfortunately, I meeted the same issue. Here is my log :

time="2024-08-12T10:04:03Z" level=debug msg="This is the watchtower container /watchtower"
time="2024-08-12T10:04:03Z" level=info msg="Session done" Failed=0 Scanned=7 Updated=0 notify=no
time="2024-08-12T10:04:03Z" level=debug msg="Scheduled next run: 2024-08-12 10:04:57 +0000 UTC"
time="2024-08-12T10:04:57Z" level=debug msg="Checking containers for updated images"
time="2024-08-12T10:04:57Z" level=debug msg="Retrieving running containers"
time="2024-08-12T10:04:57Z" level=debug msg="Trying to load authentication credentials." container=/watchtower image="<my_private_registries>/containrrr/watchtower:1.7.1"
time="2024-08-12T10:04:57Z" level=debug msg="Loaded auth credentials for user cattom, on registry <my_private_registries>, from file /config.json"
time="2024-08-12T10:04:57Z" level=debug msg="Got image name: <my_private_registries>/containrrr/watchtower:1.7.1"
time="2024-08-12T10:04:57Z" level=debug msg="Credentials loaded"
time="2024-08-12T10:04:57Z" level=debug msg="Checking if pull is needed" container=/watchtower image="<my_private_registries>/containrrr/watchtower:1.7.1"
time="2024-08-12T10:04:57Z" level=debug msg="Built challenge URL" URL="https://<my_private_registries>/v2/"
time="2024-08-12T10:04:57Z" level=debug msg="Got response to challenge request" header="Basic realm=\"restricted\"" status="401 Unauthorized"
time="2024-08-12T10:04:57Z" level=debug msg="Parsing image ref" host=<my_private_registries> image=containrrr/watchtower normalized=<my_private_registries>/containrrr/watchtower tag=1.7.1
time="2024-08-12T10:04:57Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://<my_private_registries>/v2/containrrr/watchtower/manifests/1.7.1"
time="2024-08-12T10:04:58Z" level=debug msg="Found a remote digest to compare with" remote="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038"
time="2024-08-12T10:04:58Z" level=debug msg=Comparing local="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038" remote="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038"
ldexterldesign commented 2 months ago

You had one job 🙄

ciberboy commented 2 months ago

Apologies. I haven't been able to reproduce the error since I enabled debug logging. Will do as soon as it happens again

On Sat, 31 Aug 2024, 16:42 Lewis Dexter Litanzios, @.***> wrote:

You had one job 🙄

— Reply to this email directly, view it on GitHub https://github.com/containrrr/watchtower/issues/1999#issuecomment-2322938125, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJSISCCAN774OLQYVLYNGTZUHP4RAVCNFSM6AAAAABK24GRN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRSHEZTQMJSGU . You are receiving this because you commented.Message ID: @.***>

yutotakano commented 1 month ago

I encountered this issue today with Watchtower version 1.7.1. Similar relevant logs, where the digests do not match between local and remote, but Watchtower is unable to find new images after a subsequent pull.

I've had to redact the image and organization names.

time="2024-09-28T17:35:46Z" level=debug msg="Trying to load authentication credentials." container=/<image> image="ghcr.io/<redacted>/<image>:latest"
time="2024-09-28T17:35:46Z" level=debug msg="Loaded auth credentials for user <redacted-sa>, on registry ghcr.io, from file /config.json"
time="2024-09-28T17:35:46Z" level=debug msg="Got image name: ghcr.io/<redacted>/<image>:latest"
time="2024-09-28T17:35:46Z" level=debug msg="Credentials loaded"
time="2024-09-28T17:35:46Z" level=debug msg="Checking if pull is needed" container=/<image> image="ghcr.io/<redacted>/<image>:latest"
time="2024-09-28T17:35:46Z" level=debug msg="Built challenge URL" URL="https://ghcr.io/v2/"
time="2024-09-28T17:35:46Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:user/image:pull\"" status="401 Unauthorized"
time="2024-09-28T17:35:46Z" level=debug msg="Checking challenge header content" realm="https://ghcr.io/token" service=ghcr.io
time="2024-09-28T17:35:46Z" level=debug msg="Setting scope for auth token" image=ghcr.io/<redacted>/<image> scope="repository:<redacted>/<image>:pull"
time="2024-09-28T17:35:46Z" level=debug msg="Credentials found."
time="2024-09-28T17:35:46Z" level=debug msg="Parsing image ref" host=ghcr.io image=<redacted>/<image> normalized=ghcr.io/<redacted>/<image> tag=latest
time="2024-09-28T17:35:46Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://ghcr.io/v2/<redacted>/<image>/manifests/latest"
time="2024-09-28T17:35:47Z" level=debug msg="Found a remote digest to compare with" remote="sha256:aa91f7c79a35fce42037a2bfc8f83cba061ba53c70196372a26265f525a3bd9d"
time="2024-09-28T17:35:47Z" level=debug msg=Comparing local="sha256:04039f7a7e3a2983efdaefba3abf4813931db4534a088cd1511e007c2e28203a" remote="sha256:aa91f7c79a35fce42037a2bfc8f83cba061ba53c70196372a26265f525a3bd9d"
time="2024-09-28T17:35:47Z" level=debug msg="Digests did not match, doing a pull."
time="2024-09-28T17:35:47Z" level=debug msg="Pulling image" container=/<image> image="ghcr.io/<redacted>/<image>:latest"
time="2024-09-28T17:36:03Z" level=debug msg="No new images found for /<image>"