search

just-containers / s6-overlay

s6 overlay for containers (includes execline, s6-linux-utils & a custom init)
Other
3.78k stars 212 forks source link

Issue with S6_SERVICES_GRACETIME not taking effect for graceful shutdown #337

Closed ingluisjimenez closed 2 years ago

ingluisjimenez commented 3 years ago

Hi,

I have a CLI application in PHP which has the following run file.

#!/usr/bin/execlineb -P
with-contenv s6-setuidgid www-data 
php /app/workers/worker_runner.php --worker=commands

I am experiencing an issue where S6_SERVICES_GRACETIME is not taking effect during shutdown, and the container is receiving the second TERM signal immediately, leading to the supervised processes to being terminated immediately.

I have S6_SERVICES_GRACETIME set to 50000 and S6_KILL_GRACETIME set to 55000. Below you can notice in the timestamp of the logs, how there is only 200ms between the time when the container is "waiting for services" and then when it logs "sending all processes the TERM signal". You can also notice how S6_KILL_GRACETIME is working properly, as there are 55 seconds between the time when it sends the second TERM signal and when it finally logs "sending all processes the KILL signal and exiting", as expected.

One thing I have been thinking is whether the fact that the "supervisor" process is dying has something to do and the reason why S6_SERVICES_GRACETIME is not being taken into account?

3/25/21 3:00:24.168 PM [cont-finish.d] executing container finish scripts... 3/25/21 3:00:24.170 PM [cont-finish.d] 00-nginx.sh: executing... 3/25/21 3:00:24.177 PM [finish nginx] shutting down gracefully 3/25/21 3:00:24.292 PM [cont-finish.d] done. 3/25/21 3:00:24.293 PM [s6-finish] waiting for services. 3/25/21 3:00:24.331 PM s6-svwait: fatal: supervisor died 3/25/21 3:00:24.532 PM [s6-finish] sending all processes the TERM signal. 3/25/21 3:01:19.734 PM [s6-finish] sending all processes the KILL signal and exiting.

Thank you in advance.

skarnet commented 2 years ago

Sorry for the delay in answering. This was a known issue that was difficult to fix with the previous version of s6-overlay. Now that 3.0.0.0 is out (source only atm, pre-built tarballs to come soon), please try again; the issue should be solved.

lmagyar commented 2 years ago

It seems that there are situations where S6_SERVICES_GRACETIME is still ignored (#457) and some where it works (#458) though other issues make it still useless.

skarnet commented 2 years ago

@lmagyar No, you are experiencing issues in some situations because your mariadb run script is full of timing issues. As explained in the other thread, you need to rewrite it in order to take advantage of the supervision features.