Restart order reversed for `depends-on` containers

ptr727 commented 1 year ago

Describe the bug

I am using labels to define depends-on relationships between containers, i.e. Home Assistant depends on MariaDB.
Last night MariaDB updated, and was eager to see the dependency logic in action, else normally HA stop working.

I noticed couple things that do not seem right:

The logs show that MariaDB was stopped before Home Assistant, this should be the other way around.
- Since Home Assistant has the com.centurylinklabs.watchtower.depends-on: zigbee2mqtt,mariadb label, Home Assistant should have been stopped first, then MariaDB.
The logs state that Home Assistant was created, it should have been restarted, not recreated.
- Maybe this is just a log issues (there is no matching removing image), but big difference between stop and start, vs. stop delete recreate just for dependency logic.
As with the shutdown order, the startup order is also the wrong way round, since Home Assistant depends on MariaDB, MariaDB should be started first.
- The logs show that Home Assistant was started (created) before MariaDB, the wrong way round.

Steps to reproduce

Configure depends-on labels
Wait for an update that effects a dependency relationship
Observe the shutdown and restart order is reversed

Expected behavior

If A depends on B, then A needs to be stopped before B is stopped, and B needs to be started before A is started.

Screenshots

No response

Environment

Debian / Proxmox 7 Docker version 23.0.1, build a5ee5b1

Your logs

Home Assistant Labels:

com.centurylinklabs.watchtower.depends-on: zigbee2mqtt,mariadb
com.centurylinklabs.watchtower.enable: true

Watchtower logs:

time="2023-03-22T11:38:20-07:00" level=info msg="Watchtower 1.5.3"
time="2023-03-22T11:38:20-07:00" level=info msg="Using notifications: smtp"
time="2023-03-22T11:38:20-07:00" level=info msg="Only checking containers using enable label"
time="2023-03-22T11:38:20-07:00" level=info msg="Scheduling first run: 2023-03-23 02:00:00 -0700 PDT"

time="2023-03-25T02:01:24-07:00" level=info msg="Found new falconchristmas/fpp:latest image (d237ad831613)"
time="2023-03-25T02:03:33-07:00" level=info msg="Found new lscr.io/linuxserver/sabnzbd:latest image (01849c97250d)"
time="2023-03-25T02:04:37-07:00" level=info msg="Found new lscr.io/linuxserver/mariadb:latest image (7e80f69900cb)"
time="2023-03-25T02:05:53-07:00" level=info msg="Stopping /mariadb (eb257433468f) with SIGTERM"
time="2023-03-25T02:05:58-07:00" level=info msg="Stopping /sabnzbd (645b0ba74614) with SIGTERM"
time="2023-03-25T02:06:04-07:00" level=info msg="Stopping /fpp (61bdd0397362) with SIGTERM"
time="2023-03-25T02:07:34-07:00" level=info msg="Stopping /home-assistant (1be50d8912ec) with SIGTERM"
time="2023-03-25T02:09:07-07:00" level=info msg="Creating /home-assistant"
time="2023-03-25T02:09:09-07:00" level=info msg="Creating /fpp"
time="2023-03-25T02:09:10-07:00" level=info msg="Creating /sabnzbd"
time="2023-03-25T02:09:11-07:00" level=info msg="Creating /mariadb"
time="2023-03-25T02:09:13-07:00" level=info msg="Removing image e214e0cc9b8d"
time="2023-03-25T02:09:14-07:00" level=info msg="Removing image 6af1274c3618"
time="2023-03-25T02:09:15-07:00" level=info msg="Removing image aa188807854c"
time="2023-03-25T02:09:15-07:00" level=info msg="Session done" Failed=0 Scanned=26 Updated=3 notify=no

Email notification:

Found new falconchristmas/fpp:latest image (d237ad831613)
Found new [lscr.io/linuxserver/sabnzbd:latest](http://lscr.io/linuxserver/sabnzbd:latest) image (01849c97250d)
Found new [lscr.io/linuxserver/mariadb:latest](http://lscr.io/linuxserver/mariadb:latest) image (7e80f69900cb)
Stopping /mariadb (eb257433468f) with SIGTERM
Stopping /sabnzbd (645b0ba74614) with SIGTERM
Stopping /fpp (61bdd0397362) with SIGTERM
Stopping /home-assistant (1be50d8912ec) with SIGTERM
Creating /home-assistant
Creating /fpp
Creating /sabnzbd
Creating /mariadb
Removing image e214e0cc9b8d
Removing image 6af1274c3618
Removing image aa188807854c



### Additional context

_No response_

notDavid commented 1 year ago

Yea, I think this would probably explain my issue too...

email-notification

``` Found new qmcgaw/gluetun:latest image (f0ad3492d715) Stopping /gluetun (4522c3d81da7) with SIGTERM Stopping /freshrss (6fe297dcf3a6) with SIGTERM Creating /freshrss Error response from daemon: No such container: 4522c3d81da77a4d7387e7ddefd4cde5ae4e2934787cc51f4df0e422578b9974 Creating /gluetun Removing image 8ec582956865 ``` gluetun needs to be up and running before freshrss can be started. freshrss does not restart after a gluetun update.

DrFrankensteinUK commented 1 year ago

We are seeing this same behaviour with other stacks – For example GlueTUN VPN container. In my case named 'vpngluetun' with the label below on each of the attached containers. qBittorrent / Prowlarr/ Firefox in that order in the stack.

labels:
  com.centurylinklabs.watchtower.depends-on: "vpngluetun"

Last night's update ran in the following order

time="2023-05-08T02:00:48+01:00" level=info msg="Found new linuxserver/qbittorrent:latest image (b44efb4fb344)"
time="2023-05-08T02:00:53+01:00" level=info msg="Found new qmcgaw/gluetun:latest image (a8730831620f)"

time="2023-05-08T02:03:34+01:00" level=info msg="Stopping /vpngluetun (b206bb9bbbf2) with SIGTERM"
time="2023-05-08T02:03:38+01:00" level=info msg="Stopping /vpn-qbittorrent (628794b307bb) with SIGTERM"
time="2023-05-08T02:03:50+01:00" level=info msg="Stopping /vpn-prowlarr (5cdec1375c29) with SIGTERM"
time="2023-05-08T02:04:01+01:00" level=info msg="Stopping /firefox (590992413c43) with SIGTERM"

time="2023-05-08T02:04:08+01:00" level=info msg="Creating /firefox"
time="2023-05-08T02:04:08+01:00" level=error msg="Error response from daemon: No such container: b206bb9bbbf2be1703f854b37a558fc5c7e7380c90c15604688b3724814a5fa6"

time="2023-05-08T02:04:08+01:00" level=info msg="Creating /vpn-prowlarr"
time="2023-05-08T02:04:09+01:00" level=error msg="Error response from daemon: No such container: b206bb9bbbf2be1703f854b37a558fc5c7e7380c90c15604688b3724814a5fa6"

time="2023-05-08T02:04:09+01:00" level=info msg="Creating /vpn-qbittorrent"
time="2023-05-08T02:04:09+01:00" level=error msg="Error response from daemon: No such container: b206bb9bbbf2be1703f854b37a558fc5c7e7380c90c15604688b3724814a5fa6"

time="2023-05-08T02:04:09+01:00" level=info msg="Creating /vpngluetun"

piksel commented 1 year ago

@DrFrankensteinUK Try adding a / before the container name

labels:
  com.centurylinklabs.watchtower.depends-on: "/vpngluetun"

or try using containrrr/watchtower:latest-dev.

DrFrankensteinUK commented 1 year ago

Thanks - I spotted this suggested fix in another issue thread. I am due an update tonight so will see how it goes - Any ideas when the next stable release will be out including the fix?

DrFrankensteinUK commented 1 year ago

@DrFrankensteinUK Try adding a / before the container name
labels:
  com.centurylinklabs.watchtower.depends-on: "/vpngluetun"
or try using containrrr/watchtower:latest-dev.

OK - The update took place and in the correct order but has highlighted a difference issue where the existing containers seem to be looking for the old container ID and do not attach.. After some searching of various threads I don't think there is a sane way to allow Watchtower to update the VPN stack without issue - going to do manual updates..

notDavid commented 1 year ago

going to do manual updates..

I guess as a workaround we could use pre-update and post-update to stop/start the containers...

DrFrankensteinUK commented 1 year ago

going to do manual updates..

I guess as a workaround we could use pre-update and post-update to stop/start the containers...

For the couple of times I tested the attached container had just completely stalled with no signs of life – empty logs so not sure if this would work. - For the minimal amount of time it takes to just update the container vs time troubleshooting going to run with the latter (learning from spending hours on this kind of thing before) :)

containrrr / watchtower