Child processes are not reaped - growing number of zombie processes

[vlcinsky]

We used dockerize with script, which is regularly fetching some external data.

The script is here: https://gitlab.com/tamtamresearch/cet/datahub/app-es-openlr/svc_es_openlr_doit/snippets/1908196

After a day we have found, CPU usage (in %) is growing:

• green: user
• red: system
• black: iowait

Apparently, we did one system reboot.

Researching the cause, we have found, that there are thousands of zombie processes and the number was steadily growing.

The call in our Dockerfile looked like:

ENTRYPOINT ["dockerize", "-wait", "tcp://db:5432", "./repeat.sh"]
CMD ["3m", "./handlers/downloadData.php" ]

Changing it to (note: the call I put into original issue was mistakenly simple copy of the previous one, I cannot remember exactly what was the fix, I guess we have used shell or bash with -c)

ENTRYPOINT ["dockerize", "-wait", "tcp://db:5432", "./repeat.sh"]
CMD ["3m", "./handlers/downloadData.php" ]

resolved the problem: the number of zombie processes is zero now.

Conclusions

To me it seems, like dockerize script is not taking care of reaping child processes which were terminated and due to the environment, where it runs, nothing else does this important cleanup work.

It seems we hit similar issue as author of pull request #126

[JamesJJ]

Hi @vlcinsky , I understand that including process reaping in dockerize is desirable in some cases, however I'd suggest considering using tini as your ENTRYPOINT for example something like this:

ENTRYPOINT ["/sbin/tini", "-g", "--"]
CMD ["dockerize", "-wait", "tcp://db:5432", "./repeat.sh", "3m", "./handlers/downloadData.php" ]

tini is dedicated to handling reaping zombie processes, and I've found it works without any problems in production.

[vlcinsky]

Thanks @JamesJJ for nice hint.

Anyway word "some" in

process reaping in dockerize is desirable in some cases

sounds weird to me.

Utility starting processes within docker shall do process cleanup.

Resolutions are at least two:

• modify dockerize to take care of process reaping
• or mention in dockerize the fact, processes shall be reaped and tiny is one of possible solutions.

[fiendish]

The call in our Dockerfile looked like:

ENTRYPOINT ["dockerize", "-wait", "tcp://db:5432", "./repeat.sh"]
CMD ["3m", "./handlers/downloadData.php" ]

Changing it to

ENTRYPOINT ["dockerize", "-wait", "tcp://db:5432", "./repeat.sh"]
CMD ["3m", "./handlers/downloadData.php" ]

resolved the problem

These look the same to me. What am I missing?

[vlcinsky]

@fiendish you are right, I put a note there, I cannot remember exact fix, but it was related to reaping and possibly using bash or sh with -c.

I would close this issue as improperly reported, but the core cause (dockerize not reaping child processes) still holds true).

[mwmahlberg]

@vlcinsky I think this is a use case problem. The way I see it, dockerize is there to start one single process when conditions are met and provides configuration based on templates.

What you are trying to achieve is what looks like a sidecar container for a k8s pod. If that is the case, you might want to have a deep look into k8s cronjobs.

Furthermore, with repeat.sh you are pretty much reinventing the wheel. Using a simple cron expression, you can have your script run and even child processes reaped if you use a cron-container as a base. I have quickly set up one for demonstration purposes. You can find it at https://github.com/mwmahlberg/crontainer

[vlcinsky]

@mwmahlberg I will think of it.

Anyway, we do use docker swarm where is no sidecar at hand.

I also would like to see log entries created by the script. I am afraid with using cron, these get hidden somewhere.

[mwmahlberg]

@vlcinsky That heavily depends on the log driver you choose and how you do your logging. If you run the image on its own via

docker run --rm mwmahlberg/crontainer

, it runs a script that „logs“ a „Hello, cronrunner!“ to stdout and you can view that in the logs as usual. (You need to wait about 1 minute until you see the first entry.)

Since I perfectly understand that logs are valuable event streams, let me suggest a more sophisticated setup, though. By using one of the various log drivers for docker, you can have the logs exactly where they need to be. An application should log to stdout and let the environment choose what to do with the logs.

I strongly suggest the fluentd log driver. Simply run a fluentd service in global mode and choose one (or more!) of the output plugins coming with fluentd or one (or more) of the gazillion third party output plugins. Personally, I save the logs for review and long term storage in Influxdb (view them with Grafana, InfluxDB comes with retention policies, so they are rotated out with about minute precision), send a duplicate of all non-debug logs via the syslog output plugin to OSSIM and certain errors (yes, there are all kind of matchers) to Kafka. To make that setup as easy as possible, you should use some structured logging for the log messages.

You see, you do not need to be concerned about your logs. ;)