tini Exits Too Early Leading to Graceful Termination Failure

[jschwinger233]

TL;DR

When dealing with multi sub processes, tini is likely to return without waiting for all children to exit, and that drives kernel send SIGKILL to the remaining processes under the same pid namespace, going against the design principle for tini: allowing for signal forwarding and graceful termination.

Steps to Reproduce

0. get a tini binary

1. prepare a python script as followed:

   # ppp.py
   import os
   import time
   import signal
   pid = os.fork()
   if pid == 0:
   signal.signal(15, lambda _, __: time.sleep(1))
   cpid = os.fork()
   time.sleep(1000)

Note that I use time.sleep(1) to act as if there is some time consuming operation for graceful termination.

2. make sure the script ppp.py and tini bin tini are in the $(pwd), then run a docker container as followed:

   docker run -d --rm -v $(pwd):/src -w /src python /src/tini -g -s -- python /src/ppp.py

3. inspect process tree

   root     335889      1  0 Feb24 ?        00:05:21 /usr/bin/containerd
   root       2707 335889  0 23:02 ?        00:00:00  \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/7ea94ebea673367a6e31f136115dcbe5d0d18bc4c343ca5e834549f30ba7b189 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
   root       2747   2707  0 23:02 ?        00:00:00      \_ /src/tini -g -s -- python /src/ppp.py
   root       2810   2747  3 23:02 ?        00:00:00          \_ python /src/ppp.py
   root       2840   2810  0 23:02 ?        00:00:00              \_ python /src/ppp.py
   root       2841   2840  0 23:02 ?        00:00:00                  \_ python /src/ppp.py

4. use strace(1) to watch the last python process 2841: strace -fTp 2841, then send SIGTERM to tini: kill -15 2747

What We Expected

Since there is no multiple process group, and we specified the tini with -s -g, the actions for tini should have been:

1. forward SGTERM to all the python processes
2. wait for all python processes to exit (graceful termination)
3. exit as the last the process in the pid namespace

What We Got

However, we could simply observed that the second (2840) and the third (2841) python processes received SIGKILL as soon as SIGTERM came, demonstrating that tini didn't wait for their exits so that their graceful termination failed.

Root Cause and Suggestions

1. Who sent the SIGKILL? Kernel

   If the "init" process of a PID namespace terminates, the kernel terminates all of the processes in the namespace via a SIGKILL signal.

2. Why tini didn't wait? https://github.com/krallin/tini/blob/b9f42a0e7bb46efea0c9e3d8610c96ab53b467f8/src/tini.c#L546-L560 Look at the second branch case 0, whose semantic in the waitpid(2) is:

   waitpid(): on success, returns the process ID of the child whose state has changed; if WNOHANG was specified and one or more child(ren) specified by pid exist, but have not yet changed state, then 0 is returned.

So waitpid(-1, NOHANG)=0 means there is no "waitable" child(ren), but there IS child(ren), and that's when tini exits: with children still alive

3. Is this expected? You might argue that, as long as the direct child process has decent behavior of handling SIGTERM, such as the first python 2810 should wait for the second python 2840 before quitting, the tini is flawless. Well that's quite true, but provided that the direct child is capable of grappling with graceful termination and so on, what's the point of installing tini as pid 1 in the container? In that case, we should run the application process as pid 1, without tini.

4. Improvement Suggestions There are many ways to tackle the issue, and the principle is as simple as NOT exit until all children are gone. The syscall waitpid(2) already offers us ability to distinguish between "there is no child" and "there is no waitable children", so we just follow the doc, and change the tini exit condition.

[jschwinger233]

My friend @Zheaoli is working on a PR to address the issue.

[vsxen]

对我来说更可怕的是容器停不掉 ex: https://www.likakuli.com/posts/docker-pod-terminating/

[Hello-Linux]

@jschwinger233 @vsxen @antonmos @zimbatm When will this problem be fixed? If tini exits early without waiting for the child process, then the function of causing tini to forward signals is meaningless

[yosifkit]

tini is only designed to wait for its direct child to exit; if that child spawned its own children, then it should be signaling/waiting for them before exiting. tini will reap zombies, but not wait explicitly for the nested children to exit (since it only knows of their existence once they become a zombie, i.e. they exited and their parent exited without waiting on them).

[kiddingl]

我也遇见了这个问题，tini执行了个bash脚本，kill tini进程后会立即停止bash脚本，内部产生的子进程无法收到信号，导致tini快速退出，子进程被强制终止 后来内部命令使用了exec执行，确实可以解决这个问题，但是exec 命令后面使用管道（exec sleep 200 ｜ ls）这种就又无法生效了