search

ohwgiles / laminar

Fast and lightweight Continuous Integration
https://laminar.ohwg.net
GNU General Public License v3.0
300 stars 55 forks source link

Support pausing the build queue #217

Open cweagans opened 2 months ago

cweagans commented 2 months ago

We're working on adopting Laminar. We're probably going to package up our Laminar + all config in a Docker container. The normal case is that that container can be started and it'll run forever. One of the things that we need is a safe update process that doesn't impact currently-running jobs. So during an upgrade (maybe we're rolling out new config or a new version of Laminar or something), we have a new container that we need to replace the currently running one with. We can handle the logistics of making that happen, but we need a way to tell Laminar to stop doing things for a minute so we can bring down the container.

The effect of this would be that any currently-running jobs would continue to run until they are complete. New jobs can still be queued, but they won't be executed until we tell Laminar to resume queue processing.

The specific mechanics of how we do this don't really matter. We kicked around the idea of touching a file that laminar is watching for (e.g. /var/lib/laminar/LAMINAR_PAUSE) and then deleting it when we're ready to resume processing. We also considered a laminarc command (which would work just fine for us).

ohwgiles commented 2 months ago

Maybe you can use contexts for this? If you set EXECUTORS=0, jobs will queue indefinitely. Laminar will automatically notice the config file change.

Keep in mind that if you actually want to restart the laminar process (you mentioned "a new version of Laminar or something") then jobs in the queue are lost (the queue is not persistent)

ferki commented 2 months ago

Keep in mind that if you actually want to restart the laminar process (you mentioned "a new version of Laminar or something") then jobs in the queue are lost (the queue is not persistent)

Perhaps saving the output of laminarc show-queued could help to requeue these jobs after the restart?

ohwgiles commented 1 week ago

@ferki in general this isn't enough, since that doesn't show the parameters of those queued jobs. I don't mind considering moving queued jobs to be stored in the database, it might enable some additional features.

@cweagans does EXECUTORS=0 solve your issue?