search

cpuguy83 / containerd-shim-systemd-v1

Apache License 2.0
57 stars 2 forks source link

Short-lived processes not handled well #1

Closed cpuguy83 closed 2 years ago

cpuguy83 commented 2 years ago

For short-lived process such as echo hello or exit 1 (often used for tests), systemd can end up reporting these as failing to start. Systemd reads the pid from the pid file but by the time it has read the pid the process has exited, which breaks the protocol of "forking" services.

It also doesn't help that systemd removes the pid file, so we can't really check that the process at least started right now.

cpuguy83 commented 2 years ago

Opened bug upstream: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1954698

cpuguy83 commented 2 years ago

This is fixed by a series of commits.

This was only really a problem for execs since runc create creates a persistent process until you start the container. For exec runc exec runs the process directly.

The gist of the fix is that the shim now uses a helper process to bootstrap the exec and creates a subreaper that waits for the exec's exit state and stores that in a json file. This subreaper only sticks around for a short time (1s?), at which time systemd should have had plenty of time to attach to the exec process.

We always read the exit state from this saved json file, which is either generated due to a quickly-exited process during startup or from the process exit handler (ExecStopPost).