Pilot process takes > 10 minutes to exit unless there is a corresponding Pilot resource

[wallrj]

I noticed this in #157 and then again when I tried to write integration tests for Pilots.

I think the problem is that on receiving SIGTERM, the process polls for 10 minutes, waiting for the shutdown hooks to run. But they never run because there isn't a Pilot resource to trigger the sync and reconcileHooks methods.

/kind bug

[munnerz]

/assign

[wallrj]

Some random thoughts copied from Slack conversations:

• There's something funky about how the generic pilot shuts down.
• It gets stuck (it seems to me) while wait.Poll ing for postStop hooks to complete.
• Also note that Cassandra images use dumb-init which we bypass by making /pilot the entrypoint.
• Our pilot may not be doing a good job as pid1
• If you could prevent the database container from being signalled directly, and if the pilot could share the process namespace of the database container, the pilot wouldn't have to be concerned with starting the database.
• I'd also like to consider whether the pilot could detect and use the original entrypoint of the image where it's been injected. (But not sure if that's going to work if the image already uses dumbinit.)
• It saw what I thought was a case of the pilot not even responding to SIGKILL. The E2E tests fail because the ES namespace failed to delete within 5min. https://github.com/jetstack/navigator/pull/168#issuecomment-351103506 . I went searching through the kubelet source code yesterday and it looks like the deletionGracePeriod is used properly. I found https://github.com/moby/moby/issues/22802 which seems to suggest that SIGKILL has no effect if the process is in state uninterruptible sleep.
• But when I tested the ES pilot and ES image with docker stop, the process seemed to exit upon kill -9. I couldn't see it ever get into uninterruptible sleep (so that's probably a red herring)

[munnerz]

This should be resolved by #183

/close