search

containrrr / watchtower

A process for automating Docker container base image updates.
https://containrrr.dev/watchtower/
Apache License 2.0
17.74k stars 822 forks source link

Erron on synology : No such image #215

Closed kindyy closed 4 years ago

kindyy commented 5 years ago

Since the last update, I get this message : level=fatal msg="Error: No such image: sha256:a6dc451b8bc1df6a70d1fbd5f11f6fe04613eb1ef31acd891f12fed709fbf01c"

alxwolf commented 5 years ago

+1

alxwolf commented 5 years ago

OK, was able to fix this by stopping all running containers, starting watchtower and then restarting all other containers.

AugustoGarcia commented 5 years ago

That doesn't really solve anything, I think. Have you checked if it is updating the containers if you do that?

simskij commented 5 years ago

Can't really help you there as I dont have the device you're running it on. If anyone with a synology could either point me to exactly what's wrong or help with a PR, I'd be happy to assist.

SoulInfernoDE commented 5 years ago

I have a synology and im using your watchtower in it. However i need to check whats going on or if i did fix it in the past already and if yes how. Getting back to you if i could debug it.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

richardholst commented 5 years ago

hi there , just found this ..I have the same issue not on a synology ...on a HP micros server running ubuntu

time="2019-07-04T10:35:38Z" level=fatal msg="Error: No such image: sha256:85fe87b5a8911fa8de3d20917df6c0b6565b0d8b12e9b7dd043c64856e2c9fb3"

any idea on how to fix this ?

UPDATE: by starting watchtower container first then all the other containers it seems to work ...but everytime I reboot I have to go in and manaully restart all the containers

SoulInfernoDE commented 5 years ago

i still have this problem with the container redis.

Reproduceable... ...someone should tell watchtower to pull :latest image for containers throwing this error...

simskij commented 5 years ago

Watchtower defaults to the latest tag if no other tag has been supplied. And as I said, I cant really help you without access to a synology device or steps to reproduce the behavior on MacOS or Linux x86/64.

SoulInfernoDE commented 5 years ago

hey, you could try xpenology as this should imitate the dsm system if you are willing to :-)

Edit: From what i've understood it defaults to the latest tag but does not advice the docker deamon to pull from the :latest tag. This seems to lead watchtower to try to pull tags only which are known or associated to the container its actually trying to update. So in our case a latest tag was not included by the docker container itself but instead sha256 'replaced' latest tag in the local docker deamon images. Conclusion is that watchtower then thinks sha256 branch is needed to be pulled instead of latest branch which watchtower can't pull because it actually doesn't exist.

Could you try to reproduce it when pulling the redis container without specifying a tag directly and letting docker decide to pull the latest tag. Then when an update is out try to let watchtower update it?

SoulInfernoDE commented 4 years ago

Okay here is a screenshot showing the synology docker gui to add more information. In this section i am searching for "redis" and these are the search results:

image

As you can see it is a central image and it is tagged with only one name: redis

there is no redis/redis or redis/latest in the central version.

Other docker images down under are tagged f.e. with: bitnami/redis or rediscommander/redis-commander What i guess is that without redis/somenamehere it will not pull the added image tag: latest Instead after it was downloaded the selected tag will be automatically swapped with the sha256.

unix0r commented 4 years ago

Seems like watchtower wants to update a container image, thats not existing anymore...

ash-4.3# docker run --name watchtowerContainrrr -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --debug time="2019-08-28T18:44:03Z" level=debug msg="Sleeping for a seconds to ensure the docker api client has been properly initialized." time="2019-08-28T18:44:04Z" level=debug msg="Retrieving running containers" time="2019-08-28T18:44:04Z" level=fatal msg="Error: No such image: sha256:97c293909a272ec08df8e27dcc9707946295c03e9334812367e141291dc2256b" ash-4.3# docker image list --no-trunc REPOSITORY TAG IMAGE ID CREATED SIZE openhab/openhab 2.5.0.M1 sha256:31a4e2473c2faf70c0b496d64d1a20b63eaf70e9a9ff9f1b7c9eeb861c24ba18 7 weeks ago 525MB nextcloud latest sha256:9f69908039fd3796618deb4e0d76599677a19fc68c732bdca964d2768693ebd7 7 weeks ago 623MB wordpress latest sha256:a133f1644d54ea58d46b22636df47a44b088c2193ae48418a7754ccb36ac8fd1 7 weeks ago 451MB linuxserver/sabnzbd latest sha256:9fe7b19a0125bc1f554c0c5fde6711377667900196324ffdc8b86fe60cf2002c 7 weeks ago 229MB redis latest sha256:bb0ab8a99fe694e832e56e15567c83dee4dcfdd321d0ad8ab9bd64d82d6a6bfb 7 weeks ago 95MB mariadb latest sha256:3a2ef06682acfd0b8155e2df602b91c265f1e45d6d47ff17b146608a3d756b90 7 weeks ago 356MB containrrr/watchtower latest sha256:84803293c0e3e23d5897e18ac7a17f825f5fd4b07af934c7268a504627a10592 8 weeks ago 14.5MB linuxserver/couchpotato latest sha256:38137ec80eae019d2ec1320c32924799e7136b1a2b05584b208f97fb47ba9aa3 3 months ago 139MB jordan/icinga2 latest sha256:f63f6d431428d19a1f5f9c9e24c9d3910f92a9964d99f7f603816d6d7f616a87 3 months ago 731MB namshi/smtp latest sha256:f2dffd734243f5233b3c74808868a4166e3b05c465f6343fb3c8bc30bd72af38 6 months ago 174MB phpmyadmin/phpmyadmin latest sha256:c6ba363e7c9bba3bc96aa490e31da3e266e6f7e5d8c525fb8a36df2544c2aa54 7 months ago 166MB v2tec/watchtower latest sha256:3069a9fb302acfc664970a4cdb962b9915f15aeb089b6216f17edcd29d5dfc80 18 months ago 9.49MB

I think i deleted some old images, maybe I'll check for the missing hash and find its image for downloading it again.

thelittlefireman commented 4 years ago

Same issue for me on xpenology ```time="2019-10-03T09:43:19Z" level=fatal msg="Error: No such image: sha256:926188bc```` i try to pull latest image but same issue

UPDATE: by starting watchtower container first then all the other containers it seems to work ...but everytime I reboot I have to go in and manaully restart all the containers

It works but realy painfull

thelittlefireman commented 4 years ago

it's seems to be due to container that are not present on dockerhub or on remote registry. I make custom container with jenkins on CI and it's seems to be only with them that i've got the error. Anyone can confirm ?

jankete commented 4 years ago

Is there anything new on this? I can confirm the problem not only occurs on Synology but also on Ubuntu.

tsubus commented 4 years ago

having the same issue on my arch server, quite annoying

tammert commented 4 years ago

@jankete, @tsubus: do you have a clear reproduction path for this issue? I'd like to look at it, and I have both an Arch server and a Synology NAS, but I've been unable to replicate the issue on either machine.

jankete commented 4 years ago

In my case, I had some issues with the docker-images auf homeassistant. After rebuilding them and the watchtower container as well and excluding the homeassistant images from being updated with watchtower, my issues were gone.

nyfrk commented 4 years ago

@tammert #495 describes a reproduction path (although its reporting a different issue, the steps reported are the same that leads to this problem)

mtoensing commented 4 years ago

Ok, now I have the same problem on my Synology NAS and I am unable to fix this. I delete the watchtower container and image and stopped all containers but no luck. =/ Any workaround? I use the "run-once" option that worked flawless without a problem on my DS1019. Maybe the recent update of the docker app on Synology is the problem?

I used this command:

$ docker run -d --network host --name watchtower-once -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --cleanup --include-stopped --run-once

If I run this command it seems to start:

docker run -d --network host --name watchtower-once -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once

But that does not included the stopped containers. If I include stopped containers it runs in to that problem. Is there some kind of cache I can delete to make it work again?

joelkay commented 4 years ago

I had the same error, managed to get it working again by doing a system prune and restarting

docker system prune

mtoensing commented 4 years ago

@joelkay Will this delete all my configured containers?

joelkay commented 4 years ago

It only removes unused data. It didn't remove any of my active containers just the unused ones including the one watchtower was complaining about. Check the link for more info https://docs.docker.com/engine/reference/commandline/system_prune/

mtoensing commented 4 years ago
  1. Ok, I backed up my containers that were not running (overviewer, etc)
  2. Then I ran docker system prune
  3. Restarted the NAS
  4. Ran this: docker run -d --network host --name watchtower-once -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --cleanup --include-stopped --run-once
  5. Started the container. It worked.
  6. Ran it again: Same error. Different hash. =(
ochealaaf commented 4 years ago

The reason for this kind of error can be any running container based on an image no longer available on dockerhub. The error message contains the ImageID you need to identify the container to recreate. You may use the command

docker ps --filter ancestor=ImageID

to do so. If you are using Portainer there is an option to recreate this container directly, otherwise you have to do it manually.

mtoensing commented 4 years ago

@ochealaaf How do I find out which container name is the one missing on github? Is there a command?

ochealaaf commented 4 years ago

Sorry, I forgot - the error message contains the Digest, something like sha256:a6dc451b8bc1df6a70d1fbd5f11f6fe04613eb1ef31acd891f12fed709fbf01c

Now the command

docker images --digests

prints a list where you can hopefully find the Digest from your error message and in the same line the corresponding ImageID you need to identify the container by entering

docker ps --filter ancestor=ImageID

mtoensing commented 4 years ago

docker images --digests

lists no image with the hash that watchtower ist complaining about. This really gets very annoying.


REPOSITORY                        TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
marctv/minecraft-overviewer       latest              sha256:48bb38b7414ee65b99705aa8a0e22c1cf5e22f0ab61dcf48dfb6b4f1c6d1ec90   811d59577674        3 days ago          547MB
marctv/minecraft-bedrock-server   latest              sha256:02101a643ae08f1f5e01afd745042e8bdc51846f185dfc68490c047734183422   e049b82df3b6        3 days ago          1.83GB
marctv/minecraft-papermc-server   latest              sha256:d02fe478ecabf15e5e0c2ff28a7db2957b71c0de801b204c16c2ca43c9d6cd9e   83435bb47181        3 days ago          691MB
containrrr/watchtower             latest              sha256:079338e0af66798b561b64ef9a828c5d8424056bd82c08fbc896ddec975a1dce   8d8656e8449f        4 days ago          15.2MB
mariadb                           latest              sha256:6401937141e2eaa95ac10b6a254f46628e4ae647fbf61e47ce4bab094ebf737d   ac9c11a18222        4 days ago          357MB
mariadb                           latest              sha256:ade8a0e295a3c098e471fca9ca015df6b9c08c9b63cb6ac701d0f917a89c0a7b   ac9c11a18222        4 days ago          357MB
grafana/grafana                   latest              sha256:2533fd6c262c5ba8303c7729850db39aaffc07c2d6bc7fea0081e88480f8db27   465c4af545b1        7 days ago          158MB
php                               7-apache            sha256:138fb55ba6d15771770c6855497ef262ad36589f568ec66d3ff004d2bd4b8aba   b7ec96dce970        2 weeks ago         414MB
php                               7-apache            sha256:3ca03fadcb5bfea4dfd08f16c0e9180aaae7479525cc5201f9397bcfce4c64fb   b7ec96dce970        2 weeks ago         414MB
rednoah/filebot                   watcher             sha256:4b3b184e7696fe41d2ee579cae3c2d09e9fa2ddf174797794784464dae7c16ec   0a5d367ba08f        6 weeks ago         645MB
rednoah/filebot                   latest              sha256:defd7107283006500e8d495bdbce406e6b5bd071a1cdf33f6ecf2d05d1782ead   191de5ff555a        6 weeks ago         645MB
marctv/minecraftbedrockserver     latest              sha256:bc65a1d2015870844a78ffbc1d203837a9b331c9ae99df4f31988402030ca71c   c6ffaeb9d672        2 months ago        360MB
marctv/minecraft-overviewer       <none>              sha256:20c093dc65e056dc75696060f5062aead555987e8da7529ccaf73459cceeaf07   c2eb8f5006fb        3 months ago        542MB
marctv/minecraft-papermc-server   <none>              sha256:f509fd596b0e9668602ae32685f19728f2add88b329d9b2f9c29886387efc048   ec0e61d444bd        4 months ago        668MB
marctv/minecraft-overviewer       <none>              sha256:cd3973a2ef4ce2682c0a436b30dec751414a37b796128e0a67a3f5fb4c821dc8   eea4d73547f0        5 months ago        561MB
marctv/minecraft-papermc-server   <none>              sha256:a483d50cee522380efd6bbe2273e6eee5a19b02622d69fb6de9070632c5a4b4b   391f84bcc898        5 months ago        668MB
marctv/minecraft-overviewer       <none>              sha256:6bd991fd5f4e163810f5b428d3f5cb01cb7cbd6feb2aeed6b8d39ffaa9c98f7c   42400b1e7d66        5 months ago        561MB
marctv/minecraft-papermc-server   <none>              sha256:208035df20d3c0309bb79b9cf2c35d2f6e795d9952d35ef43eb2445d5751fd95   8f47b999f0ea        5 months ago        668MB
marctv/minecraftbedrockserver     <none>              sha256:05827c7f988c4089d6330635b7466d50936d8ca55f6639147b68aac06508ee4d   3f08db1ad776        7 months ago        356MB
marctv/minecraft-overviewer       <none>              sha256:ab1b42aeacb4b486328375d3385379dc5e95e88b063d06eeb9e5a97154496a26   497159c71f4b        7 months ago        535MB
marctv/minecraft-papermc-server   <none>              sha256:b5e810fcbf8fa51355e5fb3f649e23a861195226d65633b33dc26212e28b0fc5   0d287cf9b36f        7 months ago        667MB
marctv/minecraft-papermc-server   <none>              sha256:8582ef7318ec85a8733a0132c2c4a96509f96aa4236dd76c4b153c5be2303593   41174e5f5f15        7 months ago        667MB
marctv/minecraft-papermc-server   <none>              sha256:2a0839861dc84641dce6c4fa960fcf1be164a1360392cfc15627ab1e567491c7   0c8b66ef8f94        8 months ago        667MB
marctv/minecraft-papermc-server   <none>              sha256:24218d47fae5f40af7913c1e84a563a605c5289e91476176f2e26820821161b3   9c05570ac862        9 months ago        667MB
marctv/minecraft-overviewer       <none>              sha256:5802100ca663e4f055aa34e61c56851643466d63275edc79e1d343b44ad2638c   792b07028670        9 months ago        522MB
plusminus/jdownloader2-headless   latest              sha256:f835b6e18ef93c1c57c352c2d91d69d3894fe96e165e513ff2615f1d2bdff9f5   d2cad0460723        10 months ago       992MB
marctv/minecraft-papermc-server   <none>              sha256:822fc115c5f3cfd19b0bd07cc8dedd31d58732f0405845c0061dda3ba940efeb   9e6e3ed134ef        10 months ago       176MB
marctv/minecraftbedrockserver     <none>              sha256:7c13494190efd44e114128c46e53f93e51b04a107c18d76f8d0f13b9ee2429e0   62cbea0a0811        10 months ago       128MB
marctv/minecraft-papermc-server   <none>              sha256:d66730a87733287da428ddc44a3f14327e2b3e4bec6e29f62593a132d072564c   838983dbd941        11 months ago       177MB
marctv/minecraft-papermc-server   <none>              sha256:7c39febcec401e43500770b296ae9f273f188182b2d6e0ac26b4898f4c57f437   ca53eb45252a        11 months ago       177MB
marctv/minecraft-papermc-server   <none>              sha256:6b02be5a9f61729505ef362218d353ceaf19f311b19d8c915502064060e39e6f   d246060a3d38        12 months ago       177MB
marctv/overviewer113              <none>              sha256:619598f7c44dd12645f4012ec61ca8d3537222d100833dbeae9f7301dc29d0b7   157c36b66cfa        15 months ago       496MB
marctv/overviewer113              <none>              sha256:1ad6713acb5abf3099ff04ab626379648ceef164645408ae93cb1d04f64ee7eb   ada05be667c7        16 months ago       495MB
marctv/minecraftpaperserver       <none>              sha256:0b4e7a03f27b9ae2a1582b4dc3e926caef88d1c3cf9ec4f8ece04655510acbfb   9d8b53ad51c5        17 months ago       187MB
<none>                            <none>              <none>

Message in watchtower: 

2020-06-02 11:04:05 | stderr | time="2020-06-02T11:04:05Z" level=info msg="Running a one time update."
2020-06-02 11:04:05 | stderr | time="2020-06-02T11:04:05Z" level=info msg="Error: No such image: sha256:51485a422994f0ff866a937f9931b9190de2a7a04af3db9b81d9042560f4d2be"

My guess: It is an old hash of watchtower itself.

ochealaaf commented 4 years ago

This is very strange indeed. Perhaps you could try

docker images --digests --all

to check if there is an orphaned image for whatever reason.

jankete commented 4 years ago

The same here, the hash watchtower is complaining about is not there.

root@***:~# docker images --digests --all
REPOSITORY                  TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
linuxserver/tvheadend       latest              sha256:cc174e2161cb1890d6c4659df8505a561e4f0b6cced6bbb836db9802311ebd47   498045b3ea7a        2 hours ago         507MB
linuxserver/nextcloud       latest              sha256:c5389020586eb8aabc5709bd6f31b379d8506a927f09e0f8602c5486d1b0381e   96e693898132        34 hours ago        364MB
linuxserver/heimdall        latest              sha256:d91e7052d0529c2edcd17511a9a174c586f7cef5eb0e745da9eb5ddab8847cd0   fba057db4bda        38 hours ago        79.6MB
linuxserver/ombi            latest              sha256:681da59862f795970b73428ea1b795fef95ce36851e6bbee2bd68e115ce11750   d3e17a998b98        2 days ago          376MB
portainer/portainer         latest              sha256:55c7614b1ad61eabc27214299c42d41bb49e5ef78238c0e5783031f041499284   cd645f5a4769        3 days ago          79.1MB
containrrr/watchtower       latest              sha256:b26dfbdda14acac2b5cc862691a0e8248f510a1671532b55dabb2a5231126800   333738857dc9        3 days ago          15.2MB
linuxserver/emby            latest              sha256:15e7d948dc167c679921d901130b778e97bb8467671b1d448138ed719f58115b   a6d7c1aca0ce        4 days ago          369MB
linuxserver/mariadb         latest              sha256:9af6aab5b0e00dd189cf76882257105076cb8e6a9cc658bd3e2a9c5b476bdda3   75b947ce724b        5 days ago          351MB
rix1337/docker-rsscrawler   latest              sha256:72c28c3625b262676054b3df092c49227c678f50411b5a3002b42f83e7ced1e9   e7a5c5539869        5 days ago          468MB
linuxserver/embystat        latest              sha256:182d4536a5b51de6bdf7d48a82f466c24a8bdc0bb6f007e663d9d99288e9ff93   daa30f85ec2d        6 days ago          337MB
emby/embyserver             latest              sha256:9b161ac1732161e82dc861ded84d71d83cc4865cf6af3e2e1f1062d87a31140e   b345cce26fcb        10 days ago         292MB
jlesage/jdownloader-2       latest              sha256:a79eb9ea048669a76b2a72b36f8b9dcd285bb5cb0f3882725c1e4c81c6fcbbea   d223b06e88ff        5 weeks ago         259MB
takealug/easyepg            latest              sha256:14af20c02084ae3d02d6f8f05ef509da7f03987618dffc7d594777b7b14851bd   c8261259dad1        4 months ago        508MB
time="2020-06-05T06:39:53+02:00" level=info msg="Running a one time update."
time="2020-06-05T06:39:53+02:00" level=debug msg="Checking containers for updated images"
time="2020-06-05T06:39:53+02:00" level=debug msg="Retrieving containers including stopped and exited"
time="2020-06-05T06:39:53+02:00" level=info msg="Error: No such image: sha256:51485a422994f0ff866a937f9931b9190de2a7a04af3db9b81d9042560f4d2be"

I had watchtower running fine with a constant list of containers/images for several times in the last weeks. But today after updating itself, the error occured. So I think @mtoensing may be right with this: "My guess: It is an old hash of watchtower itself."

unix0r commented 4 years ago

I don't think it's related to an old has of watchtower, but an old hash of one specific container. In my setup there are now 8 containers, that are running permanently.

With these 8 Container, there are no problems with watchtower, and they are updated regularly. But there is also one container (an old WordPress) that results in the problem.

Just tried it and watchtower could update all container (the bad WordPress container was off) and everything was fine. After that I just started the bad container and watchtower crashes with the error.

So for me it seems like a container with a hash version that maybe is no longer available breaks watchtower.

I haven't looked through the code, but maybe there is a function that lists all running containers and checks if the hash of the images is available on dockerhub. And if it is not found, there comes the error.

So one temporary solution, that also helped me is:

  1. Find the corrupted container by updating one by one with watchtower.
  2. Save or write down the configuration of the corrupted container.
  3. Delete the container.
  4. Recreate the container by hand with the saved configuration.
  5. Check if watchtower is working again.
mtoensing commented 4 years ago

Thank you for the analysis. Wouldn't it be a good idea to change the code in watchtower to keep going after showing the notice to the user @simskij

tammert commented 4 years ago

So I've actually been looking into this too.

Seems to me it's not actually a problem of a missing Docker hub (or whatever remote registry) image, but instead a broken local image (it shouldn't be missing, as it needs to be present for the container to run). Specifically, all logs I see posted are crashing on the first thing Watchtower does after booting: checking if there are multiple instances of Watchtower running. While doing that, it checks all running containers on the local Docker daemon and in the process tries to get the image information for those containers (https://github.com/moby/moby/blob/5ffd6778244c912de94259c9cde487d270757e2a/client/image_inspect.go#L17). If that fails, it logs the exact message as we see in all logs here: Error: No such image: sha26:<hash> (see https://github.com/moby/moby/blob/5ffd6778244c912de94259c9cde487d270757e2a/client/image_inspect.go#L20).

I have been unable to reproduce a container with a broken image locally though... if anyone has an exact way for me to get a broken image going, it would help tremendously in fixing this annoying error.

tammert commented 4 years ago

Quick update: I've figured out at least one instance of where this happens!

Deleting an image while a container based on it is running is not possible, even when using docker image rm --force. However, this is possible for a stopped container. And even after the image of a stopped container is removed, the container can still be started with docker start <name>.

In that situation, Watchtower will see a running container, but it will not be able to grab the image as it is no longer available. This results in Error: No such image.

I think I know how to create a fix or at least better handling of this situation, I'll send a PR soon @simskij .

SoulInfernoDE commented 4 years ago

@simskij can we get a new docker hub image for watchtower with this patch included please? --i tried "stable-dev" branch but its build for amd64 only? i need this watchtower docker image to be running on other systems and the stable branch seems 25 days old as described on the hub page.

simskij commented 3 years ago

@SoulInfernoDE it will be in latest with the next release, which should happen sometime real soon. latest-dev is, as you say, only amd64 at the moment, as it's only supposed to be used during development.