Kubernetes nodes not starting

[LeviticusMB]

Trying to test the kubernetes driver. Created the builder using

$ docker buildx create --driver kubernetes --driver-opt replicas=4 --name k8s-office

but when I try to build something, it fails with:

$ docker buildx build --load /Users/mb/source/Till -f /Users/mb/source/Till/docker//till-erp/Dockerfile --build-arg version=dev -t localhost:5000/till-erp:dev 
[+] Building 0.0s (1/1) FINISHED                                                                                                                                                
 => ERROR [internal] booting buildkit                                                                                                                                      0.0s
------
 > [internal] booting buildkit:
------
error while calling deploymentClient.Create for "k8s-office0": the server has asked for the client to provide credentials (post deployments.apps)

Not sure what to try next? I can create new pods just find on the command line:

$ kubectl run --image nginx test
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/test created
$ kubectl get all
NAME                        READY   STATUS    RESTARTS   AGE
pod/test-679b667858-5fnnt   1/1     Running   0          13s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   1y

NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/test   1         1         1            1           13s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/test-679b667858   1         1         1       13s

Background info:

$ export DOCKER_BUILDKIT=1
$ docker build --platform=local -o . git://github.com/docker/buildx
$ mv buildx ~/.docker/cli-plugins/docker-buildx

$ docker --version
Docker version 19.03.8, build afacb8b
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:16:51Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.6", GitCommit:"b1d75deca493a24a2f87eb1efde1a569e52fc8d9", GitTreeState:"clean", BuildDate:"2018-12-16T04:30:10Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

The server is a Rancher v2.3.5 Kubernetes cluster.

[AkihiroSuda]

Maybe you need to set KUBECONFIG explicitly

[LeviticusMB]

OK tried that (export KUBECONFIG=~/.kube/config) but i get the same error. Any flag I can use to activate some kind of debug logs?

[disaster37]

Hi,

I have the issue with rancher 2.4.5.

[mtreeman]

Hi, I'm also currious why it's not working. I'm also testing Buildx against a Rancher 2.4.5 Cluster with Kubernetes 1.17.6. I've deployed buildkitd as a statefulset by hand, is there a way to add the builder manually?

[jgrnt]

I did some digging and it appears that Rancher uses a bearer token for authentication. The kubernetes builder driver loads the kubeconfig using the docker cli context api, which unfortunately ignores the token in the kubeconfig (see https://github.com/docker/cli/issues/1852).

I have not found a workaround yet