search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.19k stars 1.16k forks source link

Buildkit should wait for builds to complete before shutting down #4090

Open Evesy opened 1 year ago

Evesy commented 1 year ago

When a SIGTERM is sent to a running buildkit daemon (being used as a remote builder) it immediately stops the server which results in any ongoing builds using that server being abruptly cancelled and failing.

Buildkit should ideally wait for any running builds to complete before shutting down; allowing graceful deployments of buildkit instances

tonistiigi commented 1 year ago

ref #4065

colinodell commented 10 months ago

Related to #3282

remidebette commented 3 months ago

Hi @colinodell, Did you ever come up with a preStop script for terminating gracefully in a kubernetes context?

Thank you

colinodell commented 2 months ago

@remidebette We did, and we just open-sourced it! https://github.com/seatgeek/buildkit-prestop-script

We would certainly prefer a solution implemented directly in buildkit! Unfortunately, we weren't able to figure that out ourselves, so we put this script together as a temporary stop-gap until this issue is resolved. I'm sure there's room for improvement, but it's been working well enough for our needs so far.

remidebette commented 2 months ago

Awesome @colinodell , that is an instant star, thank you for this

PidgeyBE commented 1 month ago

Nice work @colinodell ! I've created a small PR to make sure the logs of your script end up in the pod logs as well. Also extended the readme with an alternative approach to install the prestop hook. https://github.com/seatgeek/buildkit-prestop-script/pull/1