search

weaveworks-experiments / compare-revisions

Experimental thing for comparing what's on k8s clusters
Apache License 2.0
0 stars 0 forks source link

Nothing reaps orphaned children in the compare-revisions containers #26

Open dlespiau opened 7 years ago

dlespiau commented 7 years ago

Looking at our dev instance, it seems like there are thousands of zombie [git-remote-http] processes in the container.

It's somewhat likely to be the "usual" problem of PID 1 in the container (compare-revisions itself) not reaping orphaned containers:

  1. compare-versions execs git
  2. git execs some helper processes like git-remote-http
  3. git exits with git-remote-http still around, and that process gets re-parented to PID 1
  4. PID 1 doesn't reap git-remote-http when it finishes

A couple of things can be done:

  1. make compare-revisions reap processes
  2. run some kind of supervisor as pid 1 that does this for you, a quick search found a few things: (1, 2, ...)
jml commented 7 years ago

See also https://www.fpcomplete.com/blog/2016/10/docker-demons-pid1-orphans-zombies-signals and https://github.com/fpco/pid1#readme

I think running a supervisor as pid 1 is the likely best solution. I wonder if we do this already at weave?

bboreham commented 7 years ago

Flux uses tini https://github.com/search?q=org%3Aweaveworks+tini&type=Code