kubernetes driver can't authorize with GitLab KAS #1884

Closed droopy4096 closed 2 weeks ago

Contributing guidelines

[X] I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... the documentation does not mention anything about my problem
[X] ... there are no open or closed issues that are related to my problem

Description

Using buildx with --driver=kubernetes using GitLab Kubernetes Agent Server fails Authorization.

Expected behaviour

Authorization should succeed given that kubectl commands within the same context perform as expected.

Actual behaviour

While using buildx with --driver=kubernetes from within GitLab pipelines attempting to utilize existing cluster I get:

#1 ERROR: error for bootstrap "cngbuilder-amd64": GitLab Agent Server: Unauthorized: no valid credentials provided. Trace ID: e505502d5b250ff193d441b6ee1b2fd0
#2 [cngbuilder-arm64 internal] booting buildkit
#2 ERROR: error for bootstrap "cngbuilder-arm64": GitLab Agent Server: Unauthorized: no valid credentials provided. Trace ID: 516b84bc883ea687ccd0c85479f82eaf

Buildx version

github.com/docker/buildx v0.10.5 86bdced

Docker info

No response

Builders list

$ docker buildx inspect --bootstrap
#1 [cngbuilder-amd64 internal] booting buildkit
#1 ERROR: error for bootstrap "cngbuilder-amd64": GitLab Agent Server: Unauthorized: no valid credentials provided. Trace ID: bbb6079378d897ce5ec4370d83e1dbd8
#2 [cngbuilder-arm64 internal] booting buildkit
#2 ERROR: error for bootstrap "cngbuilder-arm64": GitLab Agent Server: Unauthorized: no valid credentials provided. Trace ID: 778e870e2131f7509f90ea2f27c1f06e
------
 > [cngbuilder-amd64 internal] booting buildkit:
------
------
 > [cngbuilder-arm64 internal] booting buildkit:
------
Name:          cngbuilder
Driver:        kubernetes
Last Activity: 2023-06-14 20:31:30 +0000 UTC
Nodes:
Name:           cngbuilder-amd64
Endpoint:       kubernetes:///cngbuilder?deployment=cngbuilder-amd64&kubeconfig=%2Fbuilds%2Fgitlab-org%2Fbuild%2FCNG.tmp%2FKUBECONFIG
Driver Options: namespace="default" nodeselector="kubernetes.io/arch=amd64"
Status:         inactive
Platforms:      linux/amd64*
Name:           cngbuilder-arm64
Endpoint:       kubernetes:///cngbuilder?deployment=cngbuilder-arm64&kubeconfig=%2Fbuilds%2Fgitlab-org%2Fbuild%2FCNG.tmp%2FKUBECONFIG
Driver Options: namespace="default" nodeselector="kubernetes.io/arch=arm64"
Status:         inactive
Platforms:      linux/arm64*

Configuration

FROM alpine
RUN echo "hello"

Build logs

No response

Additional info

Talking to GitLab KAS developers I've got this response:

no valid credentials provided means that neither did the request contain the Authorization header (usually containing the CI Job Token), nor the session cookie (used via the new Environments Dashboard UI). (see code here)

so while it's difficult to troubleshoot immediately due to complexity of stack involved it does seem like there's a good chance buildx kubernetes driver strips portion of request resulting in above failures.

More on the subject here: https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/1368#note_1430995638 (including some logs etc.)

additional information

redacted KUBECONFIG:

apiVersion: v1
clusters:
- cluster:
    server: https://kas.gitlab.com/k8s-proxy
  name: gitlab
contexts:
- context:
    cluster: gitlab
    user: agent:62272
  name: gitlab-org/distribution/infrastructure/kube-agents:buildx
current-context: gitlab-org/distribution/infrastructure/kube-agents:buildx
kind: Config
preferences: {}
users:
- name: agent:62272
  user:
     token: xxxxxx

downloading above KUBECONFIG locally I can run kubectl commands, however buildx fails:

docker buildx create --use --name cngbuilder --platform=linux/amd64 --node=cngbuilder-amd64 --driver=kubernetes --driver-opt="namespace=default,nodeselector=kubernetes.io/arch=amd64"
cngbuilder

docker buildx create --append --name cngbuilder --platform=linux/arm64 --node=cngbuilder-arm64 --driver=kubernetes --driver-opt="namespace=default,nodeselector=kubernetes.io/arch=arm64"
cngbuilder

docker buildx inspect --bootstrap

resulting in:

[+] Building 0.1s (2/2) FINISHED                                                                                   
 => ERROR [cngbuilder-amd64 internal] booting buildkit                                                        0.1s
 => ERROR [cngbuilder-arm64 internal] booting buildkit                                                        0.1s
------
 > [cngbuilder-amd64 internal] booting buildkit:
------
------
 > [cngbuilder-arm64 internal] booting buildkit:
------
Name:          cngbuilder
Driver:        kubernetes
Last Activity: 2023-06-15 17:00:41 +0000 UTC

Nodes:
Name:           cngbuilder-amd64
Endpoint:       kubernetes:///cngbuilder?deployment=cngbuilder-amd64&kubeconfig=%2Fhome%2Fdroopy4096%2Ftmp%2Fbar%2Fkubeconfig
Driver Options: namespace="default" nodeselector="kubernetes.io/arch=amd64"
Status:         inactive
Platforms:      linux/amd64*

Name:           cngbuilder-arm64
Endpoint:       kubernetes:///cngbuilder?deployment=cngbuilder-arm64&kubeconfig=%2Fhome%2Fdroopy4096%2Ftmp%2Fbar%2Fkubeconfig
Driver Options: namespace="default" nodeselector="kubernetes.io/arch=arm64"
Status:         inactive
Platforms:      linux/arm64*

1891 contains the fix for the issue encountered

@droopy4096 Feel free to close this issue if https://github.com/docker/buildx/pull/1891 solves it.

docker / buildx