search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.31k stars 4.88k forks source link

minikube addon `gcp-auth` fails during run with GitHub Actions with WIF and Service Account #19021

Open kevin-dimichel opened 4 months ago

kevin-dimichel commented 4 months ago

What Happened?

When trying to use a GitHub Action to run minikube with the gcp-auth addon, errors occur while when pulling the container images stored in private Google Artifact Repositories (GAR).

The gcp-auth addon starts and passes verification

minikube start output ```txt minikube start --wait=all * minikube v1.33.1 on Ubuntu 22.04 * Automatically selected the docker driver. Other choices: podman, none, ssh * Using Docker driver with root privileges * Starting "minikube" primary control-plane node in "minikube" cluster * Pulling base image v0.0.44 ... * Downloading Kubernetes v1.30.0 preload ... * Creating docker container (CPUs=2, Memory=3900MB) ... * Preparing Kubernetes v1.30.0 on Docker 26.1.1 ... - Generating certificates and keys ... - Booting up control plane ... - Configuring RBAC rules ... * Configuring bridge CNI (Container Networking Interface) ... * Verifying Kubernetes components... - Using image gcr.io/k8s-minikube/storage-provisioner:v5 * Enabled addons: storage-provisioner, default-storageclass > gcr.io/k8s-minikube/kicbase...: 16.31 MiB / 481.58 MiB [>] 3.39% ? p/s ? > gcr.io/k8s-minikube/kicbase...: 50.20 MiB / 481.58 MiB [] 10.42% ? p/s ? > gcr.io/k8s-minikube/kicbase...: 88.06 MiB / 481.58 MiB [] 18.29% ? p/s ? > gcr.io/k8s-minikube/kicbase...: 127.88 MiB / 481.58 MiB 26.55% 185.91 M > gcr.io/k8s-minikube/kicbase...: 169.58 MiB / 481.58 MiB 35.21% 185.91 M > gcr.io/k8s-minikube/kicbase...: 211.78 MiB / 481.58 MiB 43.98% 185.91 M > gcr.io/k8s-minikube/kicbase...: 251.42 MiB / 481.58 MiB 52.21% 187.20 M > gcr.io/k8s-minikube/kicbase...: 295.41 MiB / 481.58 MiB 61.34% 187.20 M > gcr.io/k8s-minikube/kicbase...: 339.39 MiB / 481.58 MiB 70.47% 187.20 M > gcr.io/k8s-minikube/kicbase...: 384.50 MiB / 481.58 MiB 79.84% 189.43 M > gcr.io/k8s-minikube/kicbase...: 428.96 MiB / 481.58 MiB 89.07% 189.43 M > gcr.io/k8s-minikube/kicbase...: 473.06 MiB / 481.58 MiB 98.23% 189.43 M > gcr.io/k8s-minikube/kicbase...: 481.58 MiB / 481.58 MiB 100.00% 215. * Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default minikube addons enable gcp-auth --alsologtostderr -v=5 --refresh I0603 21:48:14.125105 9276 out.go:291] Setting OutFile to fd 1 ... I0603 21:48:14.125378 9276 out.go:338] TERM=,COLORTERM=, which probably does not support color I0603 21:48:14.125390 9276 out.go:304] Setting ErrFile to fd 2... I0603 21:48:14.125396 9276 out.go:338] TERM=,COLORTERM=, which probably does not support color I0603 21:48:14.125623 9276 root.go:338] Updating PATH: /home/runner/.minikube/bin W0603 21:48:14.125748 9276 root.go:314] Error reading config file at /home/runner/.minikube/config/config.json: open /home/runner/.minikube/config/config.json: no such file or directory I0603 21:48:14.125889 9276 mustload.go:65] Loading cluster: minikube I0603 21:48:14.126261 9276 config.go:182] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.30.0 I0603 21:48:14.126283 9276 addons.go:597] checking whether the cluster is paused I0603 21:48:14.126384 9276 config.go:182] Loaded profile config "minikube": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.30.0 I0603 21:48:20.517591 9276 request.go:629] Waited for 197.286081ms due to client-side throttling, not priority and fairness, request: PUT:https://192.168.49.2:8443/api/v1/namespaces/kube-system/serviceaccounts/statefulset-controller I0603 21:48:20.717419 9276 request.go:629] Waited for 197.32371ms due to client-side throttling, not priority and fairness, request: PUT:https://192.168.49.2:8443/api/v1/namespaces/kube-system/serviceaccounts/storage-provisioner I0603 21:48:20.917804 9276 request.go:629] Waited for 197.318873ms due to client-side throttling, not priority and fairness, request: PUT:https://192.168.49.2:8443/api/v1/namespaces/kube-system/serviceaccounts/token-cleaner I0603 21:48:21.118047 9276 request.go:629] Waited for 197.309931ms due to client-side throttling, not priority and fairness, request: PUT:https://192.168.49.2:8443/api/v1/namespaces/kube-system/serviceaccounts/ttl-after-finished-controller I0603 21:48:21.317623 9276 request.go:629] Waited for 197.315498ms due to client-side throttling, not priority and fairness, request: PUT:https://192.168.49.2:8443/api/v1/namespaces/kube-system/serviceaccounts/ttl-controller I0603 21:48:21.517197 9276 request.go:629] Waited for 197.254205ms due to client-side throttling, not priority and fairness, request: PUT:https://192.168.49.2:8443/api/v1/namespaces/kube-system/serviceaccounts/validatingadmissionpolicy-status-controller I0603 21:48:21.529759 9276 ssh_runner.go:362] scp memory --> /var/lib/minikube/google_application_credentials.json (4522 bytes) I0603 21:48:21.529840 9276 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube I0603 21:48:21.543554 9276 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:32772 SSHKeyPath:/home/runner/.minikube/machines/minikube/id_rsa Username:docker} I0603 21:48:21.642283 9276 ssh_runner.go:362] scp memory --> /var/lib/minikube/google_cloud_project (20 bytes) I0603 21:48:21.659388 9276 addons.go:234] Setting addon gcp-auth=true in "minikube" I0603 21:48:21.659438 9276 host.go:66] Checking if "minikube" exists ... I0603 21:48:21.659915 9276 cli_runner.go:164] Run: docker container inspect minikube --format={{.State.Status}} I0603 21:48:21.673744 9276 ssh_runner.go:195] Run: cat /var/lib/minikube/google_application_credentials.json I0603 21:48:21.673807 9276 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube I0603 21:48:21.687360 9276 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:32772 SSHKeyPath:/home/runner/.minikube/machines/minikube/id_rsa Username:docker} I0603 21:48:21.772719 9276 out.go:177] - Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1 - Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1 I0603 21:48:21.773072 9276 out.go:177] - Using image gcr.io/k8s-minikube/gcp-auth-webhook:v0.1.2 - Using image gcr.io/k8s-minikube/gcp-auth-webhook:v0.1.2 I0603 21:48:21.773472 9276 addons.go:426] installing /etc/kubernetes/addons/gcp-auth-ns.yaml I0603 21:48:21.773489 9276 ssh_runner.go:362] scp gcp-auth/gcp-auth-ns.yaml --> /etc/kubernetes/addons/gcp-auth-ns.yaml (700 bytes) I0603 21:48:21.791554 9276 addons.go:426] installing /etc/kubernetes/addons/gcp-auth-service.yaml I0603 21:48:21.791576 9276 ssh_runner.go:362] scp gcp-auth/gcp-auth-service.yaml --> /etc/kubernetes/addons/gcp-auth-service.yaml (788 bytes) I0603 21:48:21.809127 9276 addons.go:426] installing /etc/kubernetes/addons/gcp-auth-webhook.yaml I0603 21:48:21.809149 9276 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/gcp-auth-webhook.yaml (5417 bytes) I0603 21:48:21.826267 9276 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.30.0/kubectl apply -f /etc/kubernetes/addons/gcp-auth-ns.yaml -f /etc/kubernetes/addons/gcp-auth-service.yaml -f /etc/kubernetes/addons/gcp-auth-webhook.yaml I0603 21:48:22.128603 9276 addons_gcpauth.go:190] refreshing existing pods I0603 21:48:22.129107 9276 kapi.go:59] client config for minikube: &rest.Config{Host:"https://192.168.49.2:8443/", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", UID:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/home/runner/.minikube/profiles/minikube/client.crt", KeyFile:"/home/runner/.minikube/profiles/minikube/client.key", CAFile:"/home/runner/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0 I0603 21:48:22.137842 9276 addons.go:470] Verifying addon gcp-auth=true in "minikube" I0603 21:48:22.138266 9276 out.go:177] * Verifying gcp-auth addon... * Verifying gcp-auth addon... I0603 21:48:22.139012 9276 kapi.go:59] client config for minikube: &rest.Config{Host:"https://192.168.49.2:8443/", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", UID:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/home/runner/.minikube/profiles/minikube/client.crt", KeyFile:"/home/runner/.minikube/profiles/minikube/client.key", CAFile:"/home/runner/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0 I0603 21:48:22.139212 9276 kapi.go:75] Waiting for pod with label "kubernetes.io/minikube-addons=gcp-auth" in ns "gcp-auth" ... I0603 21:48:22.141332 9276 kapi.go:86] Found 1 Pods for label selector kubernetes.io/minikube-addons=gcp-auth I0603 21:48:22.141345 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:22.641515 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:23.142541 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:23.643239 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:24.141963 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:24.642454 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:25.141594 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:25.641646 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:26.142338 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:26.642633 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:27.143003 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:27.641374 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:28.141573 9276 kapi.go:96] waiting for pod "kubernetes.io/minikube-addons=gcp-auth", current state: Pending: [] I0603 21:48:28.642010 9276 kapi.go:107] duration metric: took 6.502793658s to wait for kubernetes.io/minikube-addons=gcp-auth ... I0603 21:48:28.642578 9276 out.go:177] * Your GCP credentials will now be mounted into every pod created in the minikube cluster. * Your GCP credentials will now be mounted into every pod created in the minikube cluster. I0603 21:48:28.643093 9276 out.go:177] * If you don't want your credentials mounted into a specific pod, add a label with the `gcp-auth-skip-secret` key to your pod configuration. * If you don't want your credentials mounted into a specific pod, add a label with the `gcp-auth-skip-secret` key to your pod configuration. I0603 21:48:28.643460 9276 addons.go:197] Writing out "minikube" config to set gcp-auth=true... I0603 21:48:28.644157 9276 out.go:177] * The 'gcp-auth' addon is enabled * The 'gcp-auth' addon is enabled minikube addons enable ingress * ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub. You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS - Using image registry.k8s.io/ingress-nginx/controller:v1.10.1 - Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1 - Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1 * Verifying ingress addon... * The 'ingress' addon is enabled ```

After this, I run some tests that create a namespace per test and then Helm install into it. The gcp-auth secret is never created in the new namespace, even if I add a wait for 10 retries with a 10 second wait (100 seconds total).

In the gcp-auth namespace, the gcp-auth-* pod logs frequently show

 2024/06/03 21:48:27 creating pull secret: oauth2/google: status code 400: {
  "error": {
    "code": 400,
    "message": "Request contains an invalid argument.",
    "status": "INVALID_ARGUMENT"
  }
}

I didn't see any options in the image used by gcp-auth (https://github.com/GoogleContainerTools/gcp-auth-webhook) to enable additional logging.

The workflow uses the actions google-github-actions/auth@v2.1.3 to obtain creds from GCP using Workflow Identity Federation for a Service Account

jobs:
  integration-helm:
    permissions:
      contents: read
      id-token: write
    runs-on: ubuntu-latest
    steps:
      <TRUNCATED>
      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2.1.3
        with:
          project_id: '<MY_PROJECT>'
          service_account: '<MY_SERVICE_ACCOUNT>@<MY_PROJECT>.iam.gserviceaccount.com'
          workload_identity_provider: ${{ secrets.GOOGLE_WIP_GITHUB }}
      - name: Set up gcloud
        uses: google-github-actions/setup-gcloud@v2.1.0
      - name: List Container images in GAR
        shell: bash
        run: |
          set -x
          gcloud artifacts docker images list \
            us-central1-docker.pkg.dev/computer-vision-team/dev-docker/fiftyone-teams-api \
            --limit=2
      <TRUNCATED>

The step List Container images in GAR is succesful and outputs:

gcloud artifacts docker images list... ```shell gcloud artifacts docker images list us-central1-docker.pkg.dev/computer-vision-team/dev-docker/fiftyone-teams-api --limit=2 Listing items under project computer-vision-team, location us-central1, repository dev-docker. IMAGE DIGEST CREATE_TIME UPDATE_TIME SIZE us-central1-docker.pkg.dev/computer-vision-team/dev-docker/fiftyone-teams-api sha256:0473c2081ee3beba9f8b77658dbe38c4f10075f0b830e1df499c8f8d6376fa60 2024-04-15T16:37:46 2024-04-15T16:37:46 469758510 us-central1-docker.pkg.dev/computer-vision-team/dev-docker/fiftyone-teams-api sha256:066b62d0834e945a76cc6e2f62743b5e0a751251f8bed2fa46c8d1c90a625084 2024-02-21T15:45:15 2024-02-21T15:45:15 468193262 ```

To rule out the WIF, I created a new Workload Idenity Federation with a new provider and pool to ensure that it wasn't an issue with the OIDC trust relationship to Google Cloud using the service account.

I don't get these errors and while running locally on my macOS m2 laptop using the application default credentials of my user account.

Attach the log file

minikube_logs.txt

Operating System

Ubuntu

Driver

Docker

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 week ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten