Watchtower not restarting containers after update

[dorel14]

Describe the bug A clear and concise description of what the bug is. Hello , toi day i was surprised that traffmonetizer and repocket don't share anything , i 'have tried to run runme.sh and i 've seen that these 2 containers are recreated .

Why watchtower has not done his job ?

Device (please complete the following information):

• Type: Raspberry Pi 3 B+
• OS details: Linux raspbine
• Platform: arm64 (64bit)

• Version Raspbian Stretch

  Docker

• Docker Compose version: version v2.16.0
• Docker version: 23.0.1

To Reproduce Just let you docker container live and wait a new version of a container is published

Expected behavior Normally WatchTower is here to make the update automaticaly

Screenshots

Additional context` WatchTower logs

time="2023-03-16T21:08:12Z" level=warning msg="Could not do a head request for \"repocket/repocket:latest\", falling back to regular pull." container=/repocket image="repocket/repocket:latest"
time="2023-03-16T21:08:12Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:53214->127.0.0.11:53: i/o timeout" container=/repocket image="repocket/repocket:latest"
time="2023-03-16T21:08:49Z" level=info msg="Unable to update container \"/repocket\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:08:59Z" level=warning msg="Could not do a head request for \"proxyrack/pop:latest\", falling back to regular pull." container=/proxyrack image="proxyrack/pop:latest"
time="2023-03-16T21:08:59Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:32860->127.0.0.11:53: i/o timeout" container=/proxyrack image="proxyrack/pop:latest"
time="2023-03-16T21:09:35Z" level=info msg="Unable to update container \"/proxyrack\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:09:45Z" level=warning msg="Could not do a head request for \"bitping/bitping-node:latest\", falling back to regular pull." container=/bitping image="bitping/bitping-node:latest"
time="2023-03-16T21:09:45Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:58190->127.0.0.11:53: i/o timeout" container=/bitping image="bitping/bitping-node:latest"
time="2023-03-16T21:10:28Z" level=info msg="Unable to update container \"/bitping\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:10:38Z" level=warning msg="Could not do a head request for \"fazalfarhan01/earnapp:lite\", falling back to regular pull." container=/earnapp image="fazalfarhan01/earnapp:lite"
time="2023-03-16T21:10:38Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:42925->127.0.0.11:53: i/o timeout" container=/earnapp image="fazalfarhan01/earnapp:lite"
time="2023-03-16T21:11:18Z" level=info msg="Unable to update container \"/earnapp\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:11:28Z" level=warning msg="Could not do a head request for \"peer2profit/peer2profit_linux:latest\", falling back to regular pull." container=/peer2profit image="peer2profit/peer2profit_linux:latest"
time="2023-03-16T21:11:28Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:39565->127.0.0.11:53: i/o timeout" container=/peer2profit image="peer2profit/peer2profit_linux:latest"
time="2023-03-16T21:12:10Z" level=info msg="Unable to update container \"/peer2profit\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:12:20Z" level=warning msg="Could not do a head request for \"honeygain/honeygain:latest\", falling back to regular pull." container=/honeygain image="honeygain/honeygain:latest"
time="2023-03-16T21:12:20Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:58422->127.0.0.11:53: i/o timeout" container=/honeygain image="honeygain/honeygain:latest"
time="2023-03-16T21:12:55Z" level=info msg="Unable to update container \"/honeygain\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:13:05Z" level=warning msg="Could not do a head request for \"iproyal/pawns-cli:latest\", falling back to regular pull." container=/iproyal image="iproyal/pawns-cli:latest"
time="2023-03-16T21:13:05Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:40218->127.0.0.11:53: i/o timeout" container=/iproyal image="iproyal/pawns-cli:latest"
time="2023-03-16T21:13:42Z" level=info msg="Unable to update container \"/iproyal\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:13:52Z" level=warning msg="Could not do a head request for \"fnichol/uhttpd:latest\", falling back to regular pull." container=/webdashboard image="fnichol/uhttpd:latest"
time="2023-03-16T21:13:52Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:54894->127.0.0.11:53: i/o timeout" container=/webdashboard image="fnichol/uhttpd:latest"
time="2023-03-16T21:14:27Z" level=info msg="Unable to update container \"/webdashboard\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:14:37Z" level=warning msg="Could not do a head request for \"packetstream/psclient:latest\", falling back to regular pull." container=/packetstream image="packetstream/psclient:latest"
time="2023-03-16T21:14:37Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:58471->127.0.0.11:53: i/o timeout" container=/packetstream image="packetstream/psclient:latest"
time="2023-03-16T21:15:13Z" level=info msg="Unable to update container \"/packetstream\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-03-16T21:15:23Z" level=warning msg="Could not do a head request for \"containrrr/watchtower:latest\", falling back to regular pull." container=/watchtower image="containrrr/watchtower:latest"
time="2023-03-16T21:15:23Z" level=warning msg="Reason: Get \"https://index.docker.io/v2/\": dial tcp: lookup index.docker.io on 127.0.0.11:53: read udp 127.0.0.1:59740->127.0.0.11:53: i/o timeout" container=/watchtower image="containrrr/watchtower:latest"
time="2023-03-16T21:16:02Z" level=info msg="Unable to update container \"/watchtower\": Error response from daemon: Get \"https://registry-1.docker.io/v2/\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."

[MRColorR]

hello, as you can see there are info and warning messages only, no errors, just docker hub not responding to each web request. Container recreation is not always related to an updated image, it could be due settings/variables changes or unhealty container. My latest image update performed from WatchTower for images contained in money4band was on10/03/2023

• atm repocket app seems to be back up but is completely broken as you can read on their discord server.
• i have no problems with traffmonetizer now, if yout token is correctly saved in the .env file from the script ( and it should ) the container should run just fine

[dorel14]

Thx for your answer , i close this and will reopen if needed

[dorel14]

Hello I reopen and change the title of this issue

Yesterday i've seen that prooxyrack don't have shared anything on the dashboard

So i decided to have a look at they docker update

i get a look at my watchtower logs

docker logs --since 2023-04-05 watchtower

Here is what i saw

net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers). Proceeding to next."
time="2023-04-05T01:17:55Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T03:12:17Z" level=info msg="Found new proxyrack/pop:latest image (f0a566a7fb6c)"
time="2023-04-05T03:12:19Z" level=info msg="Stopping /proxyrack (5ab41b02e06d) with SIGTERM"
time="2023-04-05T03:12:51Z" level=info msg="Creating /proxyrack"
time="2023-04-05T03:13:11Z" level=info msg="Removing image baee8398b570"
time="2023-04-05T03:13:14Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=1 notify=no
time="2023-04-05T05:09:36Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T07:09:37Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T09:09:36Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T11:09:36Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T13:09:52Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T15:09:36Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T17:09:36Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T19:09:36Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2023-04-05T21:09:36Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no

As i did not see any start command , i've done sudo bash runme.sh

4 menu item to start apps => here are the logs

This menu item will launch all the apps using the configured .env file and the docker-compose.yaml file (Docker must be already installed and running)
Do you wish to proceed Y/N?  y
[+] Running 9/11
 ⠿ Container watchtower      Running                                                                                                                    0.0s
 ⠿ Container traffmonetizer  Running                                                                                                                    0.0s
 ⠿ Container webdashboard    Running                                                                                                                    0.0s
 ⠿ Container iproyal         Running                                                                                                                    0.0s
 ⠿ Container earnapp         Running                                                                                                                    0.0s
 ⠿ Container packetstream    Running                                                                                                                    0.0s
 ⠿ Container honeygain       Running                                                                                                                    0.0s
 ⠿ Container peer2profit     Running                                                                                                                    0.0s
 ⠿ Container repocket        Running                                                                                                                    0.0s
 ⠹ Container bitping         Recreate                                                                                                                  13.6s
 ⠹ Container proxyrack       Recreate

[MRColorR]

In your logs i see msg="Creating /proxyrack" so could you please tell me what is the exact issue you're experiencing?

The only concern for me is the various timeouts watchtower encounter, but eventually it succeeds. I don't know what is causing it , maybe dockerhub limiting the free unauthenticated polling rate , I know they does this but our polling rate should be slow enough.

[dorel14]

My proxyrack container has not restarted so that i have to run runme.sh , run the 4 menu item (start apps) to make docker recreate and restart the new container

I thought watchtower will do it

[MRColorR]

It should in fact. Next time dirt of all try to see if the container is up using docker ps -a if it's up see logs and check if one of the latest lines indicate that the device is logged in or if there's an error.

[dorel14]

OK i'l try this

[dorel14]

Some news I get same thing this week with repocket updated on docker , watchtower update it but don't restart correctly and then it stopped sharing bandwith until i restart it with the 4 menu in runme.sh

[MRColorR]

ok thank for the update :) . Make sure you are running the latest version of the project as some changes have been made by the docker team regarding valid docker compose project names which may affect the upgrade process.

Then if the issue appear again, try to provide the following informations:

• docker ps -a output
• a list of containers not working correctly
• last few lines of docker logs for each container not working correctly

this way we can try to narrow down the issue.

[dorel14]

The issue is still here Today repocket updated docker image

watchtower logs

time="2023-04-19T09:18:54Z" level=info msg="Found new repocket/repocket:latest image (350382a56ce6)"
time="2023-04-19T09:19:03Z" level=info msg="Stopping /repocket (01f5a9f297d0) with SIGTERM"
time="2023-04-19T09:19:42Z" level=info msg="Creating /repocket"
time="2023-04-19T09:20:11Z" level=info msg="Removing image 457206d0cfd1"

docker ps -a says everything is up and running

todays winning

and my device was not visible on repocket dashboard

to resolve i had to make

docker stop repocket docker rm repocket

sudo bash runme.sh and select 4 option to recreate and start repocket

magically after my device appear on the dashboard and i start win somme money

[MRColorR]

hello , maybe a good news from repocket team: "Hi everyone! New version of Repocket Windows, Mac, Linux PC (v1.1.25) and Docker (v1.1.18) is now available:

Fixed disconnection issues and updates to handle more users. "

[dorel14]

I've seen this , watchtower also but same thing as tomorrow , repocket stop sharing until i stop / rm / start

[MRColorR]

After receiving all this information and doing various checks I believe this is not a problem or money4band as the error does not occur in the code I wrote. The problem is within repocket itself because "PeerService(8308) -> Stop" appears in the repocket container logs every time this issue occurs and it seem repocket can't recover from this state until a manual restart is performed. I'm developing money4band not repocket and I can't fix this.

I don't know if this weird behavior is due to some weird unexpected interaction between watchtower and repocket. but given the setup of watchtower and its ability to work well with all other apps (and in many other contexts) I think this bug should be reported to the repocket team (pointing out the use of watchtower to update it) and fixed on their side. Since their app is relatively new maybe something in the logic of the code encounters an unexpected condition.

I'll close this until we receive information on why repocket behaves like this: I don't know how I could solve a problem that I "don't see" as it happens inside a specific app whose external configuration under normal conditions works.

As you can see this is a common issue not just for money4band's users but for all the repocket's users: