Why does s6 overlay changes the signal before forwarding it to actual process?

[ashishnetworks]

I have observed that whenever I send any signal to my docker container the signal gets changed by s6. Like if I sent SIGINTto my docker container, my process running behind the s6 received SIGHUPand SIGTERMtwo signals. I just want to understand why is this designed this way ? or is there any way to get the original single signal sent to the container just like it happens in tini?

[skarnet]

When you send a signal to the container, the process that receives it is pid 1. In the case of s6-overlay, this process is s6-svscan, that controls the supervision tree, i.e. the tree where all your services are running.

Your application is running either as a service under the supervision tree, or as an independent process if you defined it as a CMD.

When you send a signal to the container, you send it to give a generic command to a multiprocess environment, not to address a specific process in the container. If you send SIGINT, the container's pid 1 interprets it as "the user wants the container to shutdown", and will appropriately trigger the container's shutdown procedure. That includes sending SIGTERM to all the processes: SIGTERM is the generally accepted convention for "you are required to exit cleanly as soon as possible". (s6 does not send SIGHUP, except to loggers; I doubt your application is defined as a logger.)

So basically, s6-overlay manages an environment with a whole tree of processes and it is impossible to say "send that signal to a specific process" from the outside. From the outside, you can only send commands to the whole container.

tini does not manage a whole tree of services, but only one process; so it can specifically forward all the signals it receives to that one process, and simply exit when the process exits. It is a lot less powerful that what s6-overlay does, but that simplicity has the benefit of giving you more fine-grained control over what signals you can send from the outside to your application.

[ashishnetworks]

thanks @skarnet for the quick response. My observation is that whenever I send 1 signal to the container, my process receives two different signals. The process that I'm talking about is started through CMD, please look at the following Dockerfile's screenshot

Dockerfile:

We have a couple of workers running in nodejsprocess and when I send SIGINT to my container, like following screen

I see I get multiple signals on workers. Some of them are SIGTERMand some are SIGHUP. I don't see I have done anything to declare my process as logger. How does it know if some processes are logger

I will kindly requests your opinion on following points

1. No matter what signal has container received, am I suppose to always expect either of SIGHUP or SIGTERM?
2. If yes, is there any way I can request the s6-svscan to send me the original signal?
3. Any idea on why these two different signals are received for single signal to container?

I believe, this could be helpful to many other users as this is slightly strange behavior for people used to of different process initializers.

[skarnet]

If all the workers you're mentioning are launched and managed by the node process, then they are not considered "loggers" from s6's point of view. These workers belong to node, and s6 cannot control them in any way.

When you send a SIGINT to the container, s6-overlay shuts down the services it manages, then sends a SIGTERM to the remaining processes. That includes node, and any additional processes that node may have created. All the rest is node's behaviour when it receives a SIGTERM; if you see workers receiving a SIGHUP, it is likely due to node sending them a SIGHUP; s6-overlay has nothing to do with this.

If you want finer grained control, you will have to enter the container (via docker exec), find the node pid, and send it more specific commands, such as kill -INT if you want to send it a SIGINT. All s6-overlay knows is "we are being asked to shutdown", so it sends a SIGTERM to node, and that's the extent of its involvement.

[ashishnetworks]

Please don't consider this as any argument, but just to put my point, I retested the exactly same code with tini and found that whenever I send SIGTERM to container, my process only receives SIGTERM and in this case it does not receive SIGHUP.

I'm not trying to say that tini is better, because I agree s6 is a way more superior but I want to put my observation here that the SIGHUP that workers are receiving, it looks like its from s6 because without s6 we are not seeing any SIGHUP signal. Now this is for sure that nodejs is not sending any extra signals that we we were speculating ysterday

Please look at the following output which is with tini where no SIGNHUP are received .

Observation here is that for every SIGINT signal sent to container withs6-overlay, my every worker process is receiving two signals SIGHUPand SIGTERM.

I don't want to switch back to tini because it has as its own limitation of single process but this s6 surprise is also making me wonder about about the reasons.

[skarnet]

I can only reassert what I already said. s6 has no control over what node does internally.

I don't know why node behaves differently under tini and s6-overlay. If you want me to investigate this, I can, but you will have to provide me with your full Dockerfile so I can reproduce the behaviour.

[ashishnetworks]

@skarnet here is the repo with everything needed to reproduce the issue in quick commands. https://github.com/ashishnetworks/s6test

[skarnet]

With the repository you gave me, I can't see any node logs. Is there an option I need to give to node so it produces the output you've screenshotted?

In any case, I simply believe that the discrepancy you observed is simply due to s6-overlay sending a SIGTERM to node when it shuts down the container, whereas tini sends it a SIGINT because it forwards the signal it receives from the outside.

I strongly suspect that if you send a SIGTERM to the container that is running tini, you will see the exact same node behaviour that you see with the container that is running s6-overlay.

If it is the case, I see no issue here: node is allowed to react differently to different signals. And SIGTERM is the conventional signal that says "exit as soon as possible", so your s6-overlay container has the correct behaviour - and really you should be using SIGTERM, not SIGINT, to tell a container to exit. That is what docker stop does by default.

[ashishnetworks]

1. The output that I have put screenshot of is of different app which is close source code thats why I created another repo that I have sent you. The new test repo does not exactly gives you the same output but it surely tells you which signal the process has received.
2. I also thought that because s6-overlay is sending SIGTERMand not SIGINTnode might be behaving differently but when I tested it, it was not the case. I see only one signal received with tiniwhen I send the SIGTERMbut with s6-overly I receive two signals SIGTERMand SIGHUP.
3. About how to see the output of container, it should be visible in same terminal where you started the docker image. Please make sure you are not starting the container with -d flag
4. In case you want to see logs of already started container you can use command docker logs -f <containerId>

[skarnet]

Server will shudown for SIGTERM in a few seconds ...
Server completed shudown, gracefully!

No other output.

As expected, s6-overlay sends a SIGTERM to nodejs, which shuts down.

Now you will understand that I can't keep giving free support to something that 1. I don't see as a problem (your container exits cleanly no matter what), 2. has probably nothing to do with s6-overlay (this is all internal nodejs stuff), and 3. I don't have tools to investigate because apparently the only tools that produce your log outputs are closed source.

[ashishnetworks]

@skarnet Appreciate your time and efforts already spent to debug the issue. I understand the constraint. Thanks anyways.