add shutdown script for CoCalc projects

[DrXyzzy]

Just as CoCalc provides the option of running a project initialization script, allow specifying a shutdown script.

This could be useful, for example, when running a postgresql server in a pre-emptible instance.

GCP reference might be useful: Running shutdown scripts / Specifications.

[haraldschilly]

The problem I see is that the system will still shutdown quickly. Implementing this with a shutdown script properly is really hard. I think we should wait for "graceful shutdown" in K8S 1.21. That way more interesting to use, imho, because it doesn't depend on GCP specifics. It also inhibits the systemd shutdown sequence with a lock.

With graceful node shutdown, the kubelet is aware of underlying system shutdown events and can propagate these events to pods, ensuring containers can shut down as gracefully as possible.

more about the details behind: https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/#how-does-it-work

[williamstein]

@haraldschilly , I think you're possibly thinking about this problem in the wrong way. If we assume that CoCalc is stopping the project -- either by the user clicking stop/restart, or by idle timeout -- then issues involving Kubernetes are irrelevant. We can just send a message to the project called "you will shutdown soon", the project can do whatever it wants with that message, and only after 1 minute (say), will the project be shutdown. This may or may not have anything to do with Kubernetes; after all, cocalc can run in many environments that have nothing to do with Kubernetes.

[williamstein]

I do realize sometimes projects will stop for other reasons that aren't under the control of CoCalc, e.g., the machine the project is running on blows up... or pre-empts or something. But I think those should be viewed as a separate situation, and we should just make it clear that of course there is no guarantee that the shutdown script will always be run.