docker / buildx

Docker CLI plugin for extended build capabilities with BuildKit
Apache License 2.0
3.55k stars 481 forks source link

Using cache with Kubernetes driver #1323

Open Dorsosiv opened 2 years ago

Dorsosiv commented 2 years ago

circumstances

We are trying to use Kubernetes driver with buildx. we have created a deployment instance (using docker buildx create --platform amd64 --append --name docker-build-server --driver kubernetes --driver-opt loadbalance=random --driver-opt replicas=10 --driver-opt namespace=buildx --node amd64 --driver-opt limits.cpu=2 --config /<username>/buildx/config.toml --bootstrap) In the docker-bake.hcl file we have in our base target the configuration for the cache:

...
cache-to = ["type=local,dest=/tmp/cache,mode=max"]
cache-from = ["type=local,src=/tmp/cache,mode=max"]
...

We want to keep the cache at a local machine that from which we are running the docker buildx commands (cache type local directory). It worked for us, we did have a cache and it was growing in size. Note, we are trying to build about almost 60 images at the same time

problem

And here is the problem, we were using 10 replicas of pods as instances. And only sometimes the images that we built were using the cache that was saved.
Moreover, the cache was growing quite a lot, because we are trying to build about almost 60 images at the same time. But some of the images had similar layers. So on the first run of the docker buildx bake we didn't expect to use cache, but at the times afterwards. We did expected to see some usage of cache. The result was that only some , and even none of the images were using the cache although it was there.
So After some dig into the docs, we saw that for over driver types which do not have multiple instances , the cache is connected to the a specific instance and that is how it gets the cache. In case of several pods/ instances, it seems from our trial and error that the pods can only reach the cache that they have created and stored, and not the cache that the over pods have stored.

To elaborate, if I have 2 pods and pod-1 is building image-A and its saving it in the cache, when pod-2 will build image-A, it will not use the cache stored at the same local location by pod-1 but, it will write again the cache to there and will not use what pod-1 created before.

questions

Are we using the docker buildx create correctly ? Should we have a deployment with more than 1 pod as a buildx instance ? or should the deployment in k8s have 1 replica only, and we should create many deployments ? Are we using cache correctly with the k8s driver ?

maitredede commented 1 year ago

Hello, I am interested also about the caching behavior. In my case, I docker buildx create one builder per arch (amd64, arm64) with proper nodeselector (to build on native hardware). But for caching, I would prefer a private registry only used as cache. It may be a little slower, but more reliable, since I destroy/recreate builders on demand.