Add a `teardownFn` for gracefully terminating things before shutting down

[shayneczyzewski]

We already have a notion of setupFn, and I think we should add its dual, a teardownFn. This would be helpful to quiesce things and ensure a graceful shutdown. For example, now that we have jobs, it would be nice to shut the executor down before we shut the NodeJS server down.

In most cases, this means trapping SIGTERM and delaying the exit until you clean up. Note: most container platforms have a max wait timeout, often defaulting between 10-30 seconds. Our solution could be to trap this signal, and then call the user supplied function so they can do custom cleanup.

References:

• https://expressjs.com/en/advanced/healthcheck-graceful-shutdown.html
• https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace
• https://docs.docker.com/engine/reference/commandline/stop/

[Martinsos]

Interesting, makes sense!

I never thought about shutdown much, as I usually imagine that server dies suddenly and unexpectedly and we should always be ready for it :D. But it does make sense we should be thinking about it! The only tricky thing is that we can't really expect this function to execute, right? It might execute, but it is not guaranteed is it? So what would go into this shutdown function that we are ok with maybe executing, but maybe not?

[shayneczyzewski]

Yeah, we definitely cannot guarantee execution of teardownFn, but sudden and unexpected termination should ideally not be the norm either. :D In general, it can happen of course, but giving application code the chance, when possible, to gracefully terminate other things is always nice. Nearly every hosted environment offers SIGTERM before forceful shutdown for this purpose, so it is very useful in deployed environments (but also for things like shutting down a job executor to minimize the number of jobs that will get killed during execution).

[Martinsos]

Ok makes sense - so even if not guaranteed, it can still be pretty useful in practice!