search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
7.83k stars 1.09k forks source link

Add a new `freeBytes` gc policy #5079

Open jedevc opened 1 week ago

jedevc commented 1 week ago

This is just a proposal for now, the code still needs some tidying up and testing, but figured I'd share what I have for now.

We've had some issues with users having difficulty understanding how the cache policy works in dagger, typically run in container environments. Additionally, I've also had issues explaining to users how to configure this in buildkit. One of the problems is with setting a reasonable default - for example, on a development machine, where buildkit has access to the full disk, setting 10% max disk is totally reasonable. But in docker desktop, where you might expect to run a lot of builds, that same 10% max disk is completely inappropriate and far too small. In GitHub actions, we'd also want to use much more of the available space.

Actually, a useful setting would be to be able to configure at how much free disk space the GC should kick in, rather than the consumed disk space. For example, I might prefer to configure the GC to kick in when there is less that 20% total disk space free. This new metric also takes into account other applications that might consume disk space as well (now I can configure buildkit without needing to be aware of what else is going to be taking up lots of space), and make the most of the space on the machine.

This PR is a quick sketch of what this might look like, proposed under the config entry freeBytes - bikeshedding for naming welcome, no brilliant name springs to mind.

[!NOTE]

An alternative to this would be to instead have a keepTotalBytes option, that is like keepBytes, but instead of computing the total bytes as the sum of all the consumed cache records, would compute it as the total amount of disk space currently being consumed (i.e. roughly total - free).

jedevc commented 6 days ago

IMO, I think both settings should be followed - I think it would be quite useful and powerful to allow pruning at both constraints.

e.g. I might want to have keepBytes at 50GB, to consume max 50GB of cache - but if disk space is low, then I'd want it to use less, and always leave 10GB left.

Ideally, if we took this approach, we could modify the defaults for storage, and have freeBytes set to 1GB by default - just to prevent the accidental scenario where buildkit may consume all available space.

I'd argue that the performance metrics should be surfaced around progress/etc, docker desktop, dagger cloud, etc - this kind of poor performance already happens today on tooling like docker desktop when (naively) starting a docker container driver manually - the default disk space is small, and the 10% default is not great for large builds.

tonistiigi commented 6 days ago

e.g. I might want to have keepBytes at 50GB, to consume max 50GB of cache - but if disk space is low, then I'd want it to use less, and always leave 10GB left.

I guess then we need min-storage, max-storage and free. We can keep the existing storage limit as one but need to reevaluate the defaults.

jedevc commented 5 days ago

What would min-storage be in the above case? Is that the minimum amount of storage that the gc policy can consume (that doesn't make sense) :thinking:

I think we would just need max-storage and free.

tonistiigi commented 5 days ago

min-storage - amount BuildKit is always allowed to consume, even if free space constraint would say it should start deleting things. To make sure that BuildKit has some reserved storage that it can use for cache even if other apps are misbehaving. max-storage - BuildKit will not go over this amount even if there is still plenty of free space on disk.