Support zero-downtime swarm restarts

[ghost]

Originally reported by: Brent Tubbs (Bitbucket: btubbs, GitHub: btubbs)

---

Right now the swarm restart feature is pretty dumb. It's driven by Javascript in the UI, which just loops over each proc in the swarm and calls restart on each as fast as it can. In practice that means that the number of simultaneously restarting procs is bound by the number of simultaneous HTTP connections you can make. For swarms with more procs than that number, we already get zero-downtime restarts because the first procs have come back up before the last procs have been taken down. For swarms with fewer procs, we have downtime because there's a second or two where all procs are down.

We could improve the swarm restart behavior by building into the backend with a little more brains behind it.

• Create a new celery task called restart_swarm. It should accept a swarm ID.
• The restart_swarm task will identify all the procs on the swarm in the appropriate squad, and then split them into batches that will be restarted separately, so there's never a point where all procs are stopped (except in the case of a single proc swarm).
• Create new /swarms//restart/ resource in the API that will restart a swarm when you POST to it, by having one of the workers execute the restart_swarm task.
• Modify the frontend Javascript to call this new API endpoint when someone clicks the Restart All button on a swarm.

---

• Bitbucket: https://bitbucket.org/yougov/velociraptor/issue/109

[ghost]

Original comment by Brent Tubbs (Bitbucket: btubbs, GitHub: btubbs):

---

Mathias took this on during our recent sprint and has an implementation mostly ready to go.

[ghost]

Original comment by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco):

---

This issue could also be solved by implementing #74.

The proposal in #74 would

• require less code,
• invoke uptests on new procs, and
• leverage the routing infrastructure to ensure zero downtime while maintaining the same service level guaranteed by the procs in the swarm.

In my estimation, the only downside of #74 would be the inherent delay in uploading the procs.