Pushing to insecure registries was failing for me whenever I set rootless to true. I spent a lot of time debugging this and eventually found the buildkitd docs said the location of buildkitd.toml is different in rootful and rootless configurations. I created my own deployment.yaml and changed the configmap mount point from /etc/buildkit to /home/user/.config/buildkit and this solved the problem.
The following were the steps I took to reproduce the issue.
Deploy a local registry
docker run -d --restart=always -p "127.0.0.1:5000:5000" --name registry.k8s.local registry:2
The build works but the push fails due to insecure registry.
What did you expect to happen
When I run kubectl buildkit create buildkit --rootless --config /tmp/buildkitd.toml, I expect the configmap to be mounted at /home/user/.config/buildkit and the settings in my config file to take effect.
When I run kubectl build -t registry.k8s.local:5000/buildkit1 --push . I expect the push to succeed.
Environment Details:
kubectl buildkit version (use kubectl buildkit version)
What steps did you take and what happened
Pushing to insecure registries was failing for me whenever I set rootless to true. I spent a lot of time debugging this and eventually found the buildkitd docs said the location of buildkitd.toml is different in rootful and rootless configurations. I created my own deployment.yaml and changed the configmap mount point from /etc/buildkit to /home/user/.config/buildkit and this solved the problem.
The following were the steps I took to reproduce the issue.
Deploy a local registry
docker run -d --restart=always -p "127.0.0.1:5000:5000" --name registry.k8s.local registry:2
Add registry.k8s.local to /etc/hosts
127.0.0.1 registry.k8s.local ::1 registry.k8s.local
Create a rootless builder with insecure registry
cat < /tmp/buildkitd.toml
debug = true
[worker.containerd] namespace = "k8s.io"
[registry."registry.k8s.local:5000"] http = true insecure = true EOL kubectl buildkit create buildkit --rootless --config /tmp/buildkitd.toml
Build and push an image
kubectl build -t registry.k8s.local:5000/buildkit1 --push .
The build works but the push fails due to insecure registry.
What did you expect to happen
When I run
kubectl buildkit create buildkit --rootless --config /tmp/buildkitd.toml, I expect the configmap to be mounted at /home/user/.config/buildkit and the settings in my config file to take effect.When I run
kubectl build -t registry.k8s.local:5000/buildkit1 --push .I expect the push to succeed.Environment Details:
kubectl buildkit version)Client: 0.1.6 Builder: buildkitd github.com/moby/buildkit v0.11.5 252ae63bcf2a9b62777add4838df5a257b86e991
kubectl version)Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:14:10Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.15", GitCommit:"1d79bc3bcccfba7466c44cc2055d6e7442e140ea", GitTreeState:"clean", BuildDate:"2022-10-26T15:22:31Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/arm64"}
Laptop using Kind
sudo ctr versionor dockerddocker versionon one of your kubernetes worker nodes)containerd
Vote on this request