Open Dorsosiv opened 2 years ago
Hello,
I am interested also about the caching behavior.
In my case, I docker buildx create
one builder per arch (amd64, arm64) with proper nodeselector
(to build on native hardware).
But for caching, I would prefer a private registry only used as cache. It may be a little slower, but more reliable, since I destroy/recreate builders on demand.
circumstances
We are trying to use Kubernetes driver with buildx. we have created a deployment instance (using
docker buildx create --platform amd64 --append --name docker-build-server --driver kubernetes --driver-opt loadbalance=random --driver-opt replicas=10 --driver-opt namespace=buildx --node amd64 --driver-opt limits.cpu=2 --config /<username>/buildx/config.toml --bootstrap
) In thedocker-bake.hcl
file we have in ourbase
target the configuration for the cache:We want to keep the cache at a local machine that from which we are running the docker buildx commands (cache type local directory). It worked for us, we did have a cache and it was growing in size. Note, we are trying to build about almost 60 images at the same time
problem
And here is the problem, we were using 10 replicas of pods as instances. And only sometimes the images that we built were using the cache that was saved.
Moreover, the cache was growing quite a lot, because we are trying to build about almost 60 images at the same time. But some of the images had similar layers. So on the first run of the
docker buildx bake
we didn't expect to use cache, but at the times afterwards. We did expected to see some usage of cache. The result was that only some , and even none of the images were using the cache although it was there.So After some dig into the docs, we saw that for over driver types which do not have multiple instances , the cache is connected to the a specific instance and that is how it gets the cache. In case of several pods/ instances, it seems from our trial and error that the pods can only reach the cache that they have created and stored, and not the cache that the over pods have stored.
To elaborate, if I have 2 pods and pod-1 is building image-A and its saving it in the cache, when pod-2 will build image-A, it will not use the cache stored at the same local location by pod-1 but, it will write again the cache to there and will not use what pod-1 created before.
questions
Are we using the
docker buildx create
correctly ? Should we have a deployment with more than 1 pod as a buildx instance ? or should the deployment in k8s have 1 replica only, and we should create many deployments ? Are we using cache correctly with the k8s driver ?