Unable to use KubeCTL with Google Cloud CLI: gke-gcloud-auth-plugin is not installed

Description

During the last few days, suddenly all pipelines started to fail when trying to run kubectl commands. The error that was thrown is the following:

CRITICAL: ACTION REQUIRED: gke-gcloud-auth-plugin, which is needed for continued use of kubectl, was not found or is not executable. Install gke-gcloud-auth-plugin for use with kubectl by following https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
kubeconfig entry generated for q-machine-dev.
error: The gcp auth plugin has been removed.
Please use the "gke-gcloud-auth-plugin" kubectl/client-go credential plugin instead.
See https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke for further details

Running the solutions listed in the linked article do not work.

gcloud components install gke-gcloud-auth-plugin fails with the following error: You cannot perform this action because the Google Cloud CLI component manager is disabled for this installation.
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin fails with the package not being found

Current workaround is to run the following before running any kubectl commands:

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt update
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin kubectl
export USE_GKE_GCLOUD_AUTH_PLUGIN=True

Platforms affected

[X] Azure DevOps
[x] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 18.04
[X] Ubuntu 20.04
[X] Ubuntu 22.04
[ ] macOS 11
[ ] macOS 12
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link

from the downloaded logs:

Image: ubuntu-22.04 Version: 20221212.1 Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20221212.1/images/linux/Ubuntu2204-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20221212.1

Is it regression?

Yes, Image: ubuntu-22.04, Version: 20221204.2

Expected behavior

Running kubectl should work using the gke-google-auth-plugin.

Actual behavior

Fails because of the gke-google-auth-plugin not being installed.

Repro steps

Create a google project with a kubernetes cluster
Create a service account named azure-devops-service-account with access to it and generate a key json file

Create a new ubuntu-latest pipeline with a bash step:

 - task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      export USE_GKE_GCLOUD_AUTH_PLUGIN=True
      gcloud auth activate-service-account azure-devops-service-account@YOUR_GOOGLE_CLOUD_PROJECT_NAME.iam.gserviceaccount.com \
      --key-file <(echo $(GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON)  | base64 -d)
      (echo $(GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON) | base64 -d) > $HOME/.config/gcloud/application_default_credentials.json
      gcloud container clusters get-credentials --zone "europe-west3" "YOUR_KUBERNETES_CLUSTER_NAME" --project=YOUR_GOOGLE_CLOUD_PROJECT_NAME
      kubctl get pods

Create a secret with the name GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON with the content of the service account mentioned earlier.
Try to run the pipeline

Hi @chingucoding, can you try this as a workaround

jobs:
 windows:
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3
      # Runs a set of commands using the runners shell
      - name: Run a test
        run: |
         gcloud components list
         echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
         curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
         sudo apt update
         sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
         echo "##vso[task.setvariable variable=USE_GKE_GCLOUD_AUTH_PLUGIN]True"
         gcloud components list

This is my results

While it shows the plugin being present, it seems kubectl does not really care about that:

2022-12-15T17:54:55.6740412Z ##[section]Starting: Bash
2022-12-15T17:54:55.6748217Z ==============================================================================
2022-12-15T17:54:55.6748500Z Task         : Bash
2022-12-15T17:54:55.6748720Z Description  : Run a Bash script on macOS, Linux, or Windows
2022-12-15T17:54:55.6748944Z Version      : 3.201.1
2022-12-15T17:54:55.6749129Z Author       : Microsoft Corporation
2022-12-15T17:54:55.6749407Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2022-12-15T17:54:55.6749720Z ==============================================================================
2022-12-15T17:54:55.9801765Z Generating script.
2022-12-15T17:54:55.9805288Z ========================== Starting Command Output ===========================
2022-12-15T17:54:55.9814227Z [command]/usr/bin/bash /home/vsts/work/_temp/b6af3558-e3b3-42e9-9be6-274076cff3a3.sh
2022-12-15T17:55:02.4875795Z 
2022-12-15T17:55:02.4878505Z Your current Google Cloud CLI version is: 411.0.0
2022-12-15T17:55:02.4880104Z The latest available version is: 412.0.0
2022-12-15T17:55:02.4881576Z 
2022-12-15T17:55:02.5331919Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2022-12-15T17:55:02.5336549Z │                                                   Components                                                  │
2022-12-15T17:55:02.5337028Z ├──────────────────┬──────────────────────────────────────────────────────┬──────────────────────────┬──────────┤
2022-12-15T17:55:02.5337448Z │      Status      │                         Name                         │            ID            │   Size   │
2022-12-15T17:55:02.5337913Z ├──────────────────┼──────────────────────────────────────────────────────┼──────────────────────────┼──────────┤
2022-12-15T17:55:02.5338582Z │ Update Available │ Google Cloud CLI Core Libraries                      │ core                     │ 25.9 MiB │
2022-12-15T17:55:02.5339091Z │ Update Available │ gcloud Alpha Commands                                │ alpha                    │  < 1 MiB │
2022-12-15T17:55:02.5339608Z │ Update Available │ gcloud Beta Commands                                 │ beta                     │  < 1 MiB │
2022-12-15T17:55:02.5340112Z │ Not Installed    │ App Engine Go Extensions                             │ app-engine-go            │  4.2 MiB │
2022-12-15T17:55:02.5340593Z │ Not Installed    │ Appctl                                               │ appctl                   │ 21.0 MiB │
2022-12-15T17:55:02.5341098Z │ Not Installed    │ Artifact Registry Go Module Package Helper           │ package-go-module        │  < 1 MiB │
2022-12-15T17:55:02.5341623Z │ Not Installed    │ Cloud Bigtable Command Line Tool                     │ cbt                      │ 10.4 MiB │
2022-12-15T17:55:02.5342141Z │ Not Installed    │ Cloud Bigtable Emulator                              │ bigtable                 │  6.7 MiB │
2022-12-15T17:55:02.5342634Z │ Not Installed    │ Cloud Datalab Command Line Tool                      │ datalab                  │  < 1 MiB │
2022-12-15T17:55:02.5343311Z │ Not Installed    │ Cloud Datastore Emulator                             │ cloud-datastore-emulator │ 35.1 MiB │
2022-12-15T17:55:02.5343849Z │ Not Installed    │ Cloud Firestore Emulator                             │ cloud-firestore-emulator │ 40.2 MiB │
2022-12-15T17:55:02.5344364Z │ Not Installed    │ Cloud Pub/Sub Emulator                               │ pubsub-emulator          │ 62.4 MiB │
2022-12-15T17:55:02.5344870Z │ Not Installed    │ Cloud Run Proxy                                      │ cloud-run-proxy          │  9.0 MiB │
2022-12-15T17:55:02.5345367Z │ Not Installed    │ Cloud SQL Proxy                                      │ cloud_sql_proxy          │  7.8 MiB │
2022-12-15T17:55:02.5346099Z │ Not Installed    │ Cloud Spanner Emulator                               │ cloud-spanner-emulator   │ 28.7 MiB │
2022-12-15T17:55:02.5346620Z │ Not Installed    │ Cloud Spanner Migration Tool                         │ harbourbridge            │ 18.1 MiB │
2022-12-15T17:55:02.5347330Z │ Not Installed    │ Google Container Registry's Docker credential helper │ docker-credential-gcr    │  1.8 MiB │
2022-12-15T17:55:02.5347851Z │ Not Installed    │ Kustomize                                            │ kustomize                │  4.3 MiB │
2022-12-15T17:55:02.5348325Z │ Not Installed    │ Log Streaming                                        │ log-streaming            │ 13.9 MiB │
2022-12-15T17:55:02.5348803Z │ Not Installed    │ Minikube                                             │ minikube                 │ 31.5 MiB │
2022-12-15T17:55:02.5349270Z │ Not Installed    │ Nomos CLI                                            │ nomos                    │ 25.2 MiB │
2022-12-15T17:55:02.5349791Z │ Not Installed    │ On-Demand Scanning API extraction helper             │ local-extract            │ 13.4 MiB │
2022-12-15T17:55:02.5350281Z │ Not Installed    │ Skaffold                                             │ skaffold                 │ 20.1 MiB │
2022-12-15T17:55:02.5350777Z │ Not Installed    │ Terraform Tools                                      │ terraform-tools          │ 53.3 MiB │
2022-12-15T17:55:02.5351268Z │ Not Installed    │ anthos-auth                                          │ anthos-auth              │ 20.4 MiB │
2022-12-15T17:55:02.5351748Z │ Not Installed    │ config-connector                                     │ config-connector         │ 56.7 MiB │
2022-12-15T17:55:02.5352257Z │ Not Installed    │ gcloud app Java Extensions                           │ app-engine-java          │ 63.9 MiB │
2022-12-15T17:55:02.5352786Z │ Not Installed    │ gcloud app Python Extensions                         │ app-engine-python        │  8.6 MiB │
2022-12-15T17:55:02.5353349Z │ Not Installed    │ gcloud app Python Extensions (Extra Libraries)       │ app-engine-python-extras │ 26.4 MiB │
2022-12-15T17:55:02.5353924Z │ Not Installed    │ gke-gcloud-auth-plugin                               │ gke-gcloud-auth-plugin   │  7.6 MiB │
2022-12-15T17:55:02.5354419Z │ Not Installed    │ kpt                                                  │ kpt                      │ 12.3 MiB │
2022-12-15T17:55:02.5354891Z │ Not Installed    │ kubectl                                              │ kubectl                  │  < 1 MiB │
2022-12-15T17:55:02.5355348Z │ Not Installed    │ kubectl-oidc                                         │ kubectl-oidc             │ 20.4 MiB │
2022-12-15T17:55:02.5355807Z │ Not Installed    │ pkg                                                  │ pkg                      │          │
2022-12-15T17:55:02.5356267Z │ Installed        │ BigQuery Command Line Tool                           │ bq                       │  1.6 MiB │
2022-12-15T17:55:02.5356770Z │ Installed        │ Bundled Python 3.9                                   │ bundled-python3-unix     │ 62.2 MiB │
2022-12-15T17:55:02.5357341Z │ Installed        │ Cloud Storage Command Line Tool                      │ gsutil                   │ 15.5 MiB │
2022-12-15T17:55:02.5357852Z │ Installed        │ Google Cloud CRC32C Hash Tool                        │ gcloud-crc32c            │  1.2 MiB │
2022-12-15T17:55:02.5358438Z └──────────────────┴──────────────────────────────────────────────────────┴──────────────────────────┴──────────┘
2022-12-15T17:55:02.5358777Z To install or remove components at your current SDK version [411.0.0], run:
2022-12-15T17:55:02.5359134Z   $ gcloud components install COMPONENT_ID
2022-12-15T17:55:02.5359445Z   $ gcloud components remove COMPONENT_ID
2022-12-15T17:55:02.5359581Z 
2022-12-15T17:55:02.5359872Z To update your SDK installation to the latest version [412.0.0], run:
2022-12-15T17:55:02.5360187Z   $ gcloud components update
2022-12-15T17:55:02.5360317Z 
2022-12-15T17:55:02.6168095Z deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main
2022-12-15T17:55:02.6248701Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-12-15T17:55:02.6249374Z                                  Dload  Upload   Total   Spent    Left  Speed
2022-12-15T17:55:02.6249705Z 
2022-12-15T17:55:02.7526810Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
2022-12-15T17:55:02.8028067Z 
2022-12-15T17:55:02.8029905Z 100  2426  100  2426    0     0  13697      0 --:--:-- --:--:-- --:--:-- 13706
2022-12-15T17:55:02.8378752Z OK
2022-12-15T17:55:02.8858447Z 
2022-12-15T17:55:02.8861288Z WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
2022-12-15T17:55:02.8861675Z 
2022-12-15T17:55:03.0080952Z Hit:1 http://azure.archive.ubuntu.com/ubuntu jammy InRelease
2022-12-15T17:55:03.0119386Z Get:2 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease [114 kB]
2022-12-15T17:55:03.0120489Z Get:3 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease [99.8 kB]
2022-12-15T17:55:03.0121208Z Get:4 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
2022-12-15T17:55:03.0459359Z Get:5 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease [10.5 kB]
2022-12-15T17:55:03.1286953Z Hit:6 https://ppa.launchpadcontent.net/ubuntu-toolchain-r/test/ubuntu jammy InRelease
2022-12-15T17:55:03.2464605Z Get:7 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [769 kB]
2022-12-15T17:55:03.2524433Z Get:8 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [172 kB]
2022-12-15T17:55:03.2546725Z Get:9 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [11.5 kB]
2022-12-15T17:55:03.2586669Z Get:10 http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [498 kB]
2022-12-15T17:55:03.2635793Z Get:11 http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [76.3 kB]
2022-12-15T17:55:03.2705260Z Get:12 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [766 kB]
2022-12-15T17:55:03.2742742Z Get:13 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [130 kB]
2022-12-15T17:55:03.2783609Z Get:14 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [14.2 kB]
2022-12-15T17:55:03.4347720Z Get:15 http://azure.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [530 kB]
2022-12-15T17:55:03.4413107Z Get:16 http://azure.archive.ubuntu.com/ubuntu jammy-security/main Translation-en [114 kB]
2022-12-15T17:55:03.4907336Z Get:17 http://azure.archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [7388 B]
2022-12-15T17:55:03.4971744Z Get:18 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [457 kB]
2022-12-15T17:55:03.5007708Z Get:19 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [69.9 kB]
2022-12-15T17:55:03.5065990Z Get:20 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [622 kB]
2022-12-15T17:55:03.5118338Z Get:21 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [82.9 kB]
2022-12-15T17:55:03.5132721Z Get:22 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [11.0 kB]
2022-12-15T17:55:03.5543279Z Get:23 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages [64.9 kB]
2022-12-15T17:55:03.5685837Z Get:24 https://packages.cloud.google.com/apt cloud-sdk InRelease [6751 B]
2022-12-15T17:55:03.7816869Z Get:25 https://packages.cloud.google.com/apt cloud-sdk/main amd64 Packages [368 kB]
2022-12-15T17:55:08.4120686Z Fetched 5105 kB in 1s (4693 kB/s)
2022-12-15T17:55:09.1192045Z Reading package lists...
2022-12-15T17:55:09.2401198Z Building dependency tree...
2022-12-15T17:55:09.2411719Z Reading state information...
2022-12-15T17:55:09.3142473Z 33 packages can be upgraded. Run 'apt list --upgradable' to see them.
2022-12-15T17:55:09.3681961Z Reading package lists...
2022-12-15T17:55:09.4968978Z Building dependency tree...
2022-12-15T17:55:09.4980600Z Reading state information...
2022-12-15T17:55:09.6715804Z The following NEW packages will be installed:
2022-12-15T17:55:10.2092805Z   google-cloud-sdk-gke-gcloud-auth-plugin
2022-12-15T17:55:10.2093191Z 0 upgraded, 1 newly installed, 0 to remove and 33 not upgraded.
2022-12-15T17:55:10.2093517Z Need to get 3117 kB of archives.
2022-12-15T17:55:10.2093847Z After this operation, 10.9 MB of additional disk space will be used.
2022-12-15T17:55:10.2094453Z Get:1 https://packages.cloud.google.com/apt cloud-sdk/main amd64 google-cloud-sdk-gke-gcloud-auth-plugin amd64 412.0.0-0 [3117 kB]
2022-12-15T17:55:10.5150185Z Fetched 3117 kB in 0s (7884 kB/s)
2022-12-15T17:55:10.5754811Z Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
2022-12-15T17:55:10.6107997Z (Reading database ... 
2022-12-15T17:55:10.6108587Z (Reading database ... 5%
2022-12-15T17:55:10.6108932Z (Reading database ... 10%
2022-12-15T17:55:10.6109334Z (Reading database ... 15%
2022-12-15T17:55:10.6109879Z (Reading database ... 20%
2022-12-15T17:55:10.6110218Z (Reading database ... 25%
2022-12-15T17:55:10.6110547Z (Reading database ... 30%
2022-12-15T17:55:10.6110901Z (Reading database ... 35%
2022-12-15T17:55:10.6111230Z (Reading database ... 40%
2022-12-15T17:55:10.6141151Z (Reading database ... 45%
2022-12-15T17:55:10.6141632Z (Reading database ... 50%
2022-12-15T17:55:11.3114106Z (Reading database ... 55%
2022-12-15T17:55:11.7719926Z (Reading database ... 60%
2022-12-15T17:55:11.9208353Z (Reading database ... 65%
2022-12-15T17:55:12.7826230Z (Reading database ... 70%
2022-12-15T17:55:13.7202600Z (Reading database ... 75%
2022-12-15T17:55:14.8374176Z (Reading database ... 80%
2022-12-15T17:55:15.9342930Z (Reading database ... 85%
2022-12-15T17:55:16.8888458Z (Reading database ... 90%
2022-12-15T17:55:17.8638067Z (Reading database ... 95%
2022-12-15T17:55:17.8638386Z (Reading database ... 100%
2022-12-15T17:55:17.8638701Z (Reading database ... 236921 files and directories currently installed.)
2022-12-15T17:55:17.8707765Z Preparing to unpack .../google-cloud-sdk-gke-gcloud-auth-plugin_412.0.0-0_amd64.deb ...
2022-12-15T17:55:17.8729350Z Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (412.0.0-0) ...
2022-12-15T17:55:18.2336993Z Setting up google-cloud-sdk-gke-gcloud-auth-plugin (412.0.0-0) ...
2022-12-15T17:55:18.6379134Z NEEDRESTART-VER: 3.5
2022-12-15T17:55:19.6631266Z NEEDRESTART-KCUR: 5.15.0-1024-azure
2022-12-15T17:55:19.6637971Z NEEDRESTART-KEXP: 5.15.0-1024-azure
2022-12-15T17:55:19.6638478Z NEEDRESTART-KSTA: 1
2022-12-15T17:55:22.3326216Z 
2022-12-15T17:55:22.3327244Z Your current Google Cloud CLI version is: 411.0.0
2022-12-15T17:55:22.3327756Z The latest available version is: 412.0.0
2022-12-15T17:55:22.3328152Z 
2022-12-15T17:55:22.3773717Z ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
2022-12-15T17:55:22.3774903Z │                                                   Components                                                  │
2022-12-15T17:55:22.3775613Z ├──────────────────┬──────────────────────────────────────────────────────┬──────────────────────────┬──────────┤
2022-12-15T17:55:22.3776358Z │      Status      │                         Name                         │            ID            │   Size   │
2022-12-15T17:55:22.3777007Z ├──────────────────┼──────────────────────────────────────────────────────┼──────────────────────────┼──────────┤
2022-12-15T17:55:22.3786088Z │ Update Available │ Google Cloud CLI Core Libraries                      │ core                     │ 25.9 MiB │
2022-12-15T17:55:22.3786785Z │ Update Available │ gcloud Alpha Commands                                │ alpha                    │  < 1 MiB │
2022-12-15T17:55:22.3787412Z │ Update Available │ gcloud Beta Commands                                 │ beta                     │  < 1 MiB │
2022-12-15T17:55:22.3788102Z │ Not Installed    │ App Engine Go Extensions                             │ app-engine-go            │  4.2 MiB │
2022-12-15T17:55:22.3788725Z │ Not Installed    │ Appctl                                               │ appctl                   │ 21.0 MiB │
2022-12-15T17:55:22.3789395Z │ Not Installed    │ Artifact Registry Go Module Package Helper           │ package-go-module        │  < 1 MiB │
2022-12-15T17:55:22.3790031Z │ Not Installed    │ Cloud Bigtable Command Line Tool                     │ cbt                      │ 10.4 MiB │
2022-12-15T17:55:22.3790657Z │ Not Installed    │ Cloud Bigtable Emulator                              │ bigtable                 │  6.7 MiB │
2022-12-15T17:55:22.3791281Z │ Not Installed    │ Cloud Datalab Command Line Tool                      │ datalab                  │  < 1 MiB │
2022-12-15T17:55:22.3791925Z │ Not Installed    │ Cloud Datastore Emulator                             │ cloud-datastore-emulator │ 35.1 MiB │
2022-12-15T17:55:22.3792590Z │ Not Installed    │ Cloud Firestore Emulator                             │ cloud-firestore-emulator │ 40.2 MiB │
2022-12-15T17:55:22.3793237Z │ Not Installed    │ Cloud Pub/Sub Emulator                               │ pubsub-emulator          │ 62.4 MiB │
2022-12-15T17:55:22.3793886Z │ Not Installed    │ Cloud Run Proxy                                      │ cloud-run-proxy          │  9.0 MiB │
2022-12-15T17:55:22.3794500Z │ Not Installed    │ Cloud SQL Proxy                                      │ cloud_sql_proxy          │  7.8 MiB │
2022-12-15T17:55:22.3795123Z │ Not Installed    │ Cloud Spanner Emulator                               │ cloud-spanner-emulator   │ 28.7 MiB │
2022-12-15T17:55:22.3795814Z │ Not Installed    │ Cloud Spanner Migration Tool                         │ harbourbridge            │ 18.1 MiB │
2022-12-15T17:55:22.3796489Z │ Not Installed    │ Google Container Registry's Docker credential helper │ docker-credential-gcr    │  1.8 MiB │
2022-12-15T17:55:22.3797350Z │ Not Installed    │ Kustomize                                            │ kustomize                │  4.3 MiB │
2022-12-15T17:55:22.3797959Z │ Not Installed    │ Log Streaming                                        │ log-streaming            │ 13.9 MiB │
2022-12-15T17:55:22.3798548Z │ Not Installed    │ Minikube                                             │ minikube                 │ 31.5 MiB │
2022-12-15T17:55:22.3799246Z │ Not Installed    │ Nomos CLI                                            │ nomos                    │ 25.2 MiB │
2022-12-15T17:55:22.3799867Z │ Not Installed    │ On-Demand Scanning API extraction helper             │ local-extract            │ 13.4 MiB │
2022-12-15T17:55:22.3800596Z │ Not Installed    │ Skaffold                                             │ skaffold                 │ 20.1 MiB │
2022-12-15T17:55:22.3801197Z │ Not Installed    │ Terraform Tools                                      │ terraform-tools          │ 53.3 MiB │
2022-12-15T17:55:22.3801809Z │ Not Installed    │ anthos-auth                                          │ anthos-auth              │ 20.4 MiB │
2022-12-15T17:55:22.3802447Z │ Not Installed    │ config-connector                                     │ config-connector         │ 56.7 MiB │
2022-12-15T17:55:22.3803072Z │ Not Installed    │ gcloud app Java Extensions                           │ app-engine-java          │ 63.9 MiB │
2022-12-15T17:55:22.3803725Z │ Not Installed    │ gcloud app Python Extensions                         │ app-engine-python        │  8.6 MiB │
2022-12-15T17:55:22.3804393Z │ Not Installed    │ gcloud app Python Extensions (Extra Libraries)       │ app-engine-python-extras │ 26.4 MiB │
2022-12-15T17:55:22.3805029Z │ Not Installed    │ kpt                                                  │ kpt                      │ 12.3 MiB │
2022-12-15T17:55:22.3805615Z │ Not Installed    │ kubectl                                              │ kubectl                  │  < 1 MiB │
2022-12-15T17:55:22.3806213Z │ Not Installed    │ kubectl-oidc                                         │ kubectl-oidc             │ 20.4 MiB │
2022-12-15T17:55:22.3806796Z │ Not Installed    │ pkg                                                  │ pkg                      │          │
2022-12-15T17:55:22.3807369Z │ Installed        │ BigQuery Command Line Tool                           │ bq                       │  1.6 MiB │
2022-12-15T17:55:22.3808013Z │ Installed        │ Bundled Python 3.9                                   │ bundled-python3-unix     │ 62.2 MiB │
2022-12-15T17:55:22.3808631Z │ Installed        │ Cloud Storage Command Line Tool                      │ gsutil                   │ 15.5 MiB │
2022-12-15T17:55:22.3809265Z │ Installed        │ Google Cloud CRC32C Hash Tool                        │ gcloud-crc32c            │  1.2 MiB │
2022-12-15T17:55:22.3809897Z │ Installed        │ gke-gcloud-auth-plugin                               │ gke-gcloud-auth-plugin   │  7.6 MiB │
2022-12-15T17:55:22.3810517Z └──────────────────┴──────────────────────────────────────────────────────┴──────────────────────────┴──────────┘
2022-12-15T17:55:22.3811013Z To install or remove components at your current SDK version [411.0.0], run:
2022-12-15T17:55:22.3811472Z   $ gcloud components install COMPONENT_ID
2022-12-15T17:55:22.3811882Z   $ gcloud components remove COMPONENT_ID
2022-12-15T17:55:22.3812173Z 
2022-12-15T17:55:22.3812567Z To update your SDK installation to the latest version [412.0.0], run:
2022-12-15T17:55:22.3812998Z   $ gcloud components update
2022-12-15T17:55:22.3813224Z 
2022-12-15T17:55:23.3106565Z Activated service account credentials for: [service-account@test-project.iam.gserviceaccount.com]
2022-12-15T17:55:24.3584032Z Fetching cluster endpoint and auth data.
2022-12-15T17:55:24.5245258Z kubeconfig entry generated for q-machine-dev.
2022-12-15T17:55:24.8127177Z error: The gcp auth plugin has been removed.
2022-12-15T17:55:24.8128477Z Please use the "gke-gcloud-auth-plugin" kubectl/client-go credential plugin instead.
2022-12-15T17:55:24.8129429Z See https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke for further details
2022-12-15T17:55:24.8219017Z ##[error]Bash exited with code '1'.
2022-12-15T17:55:24.8291736Z ##[section]Finishing: Bash

I have been getting similar errors.

I had builds work as expected when they ran on runner-image version 20221204.2 recently. But all builds that ran in the version 20221212.1 failed with kubectl complaining about not finding the gke-gcloud-auth-plugin.

Hi @Jonathas-Conceicao @chingucoding , after some investigation I found out that issue started when they updated Google Cloud CLI to 4.11 on December 6th which was introduced to this image 20221204.2 also. Additionally I found out when we install auth plug-in google-cloud-sdk-gke-gcloud-auth-plugin version is (412.0.0-0) it has newest version and google-cloud-sdk on image is 4.11, which probably causing issue in this case. Please update google-cloud-sdk as well google-cloud-sdk-gke-gcloud-auth-plugin version (412.0.0-0), here is a code

jobs:
 windows:
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3
      # Runs a set of commands using the runners shell
      - name: Run a test
        run: |
         gcloud components list
         echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
         curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
         sudo apt update
         sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
         echo "##vso[task.setvariable variable=USE_GKE_GCLOUD_AUTH_PLUGIN]True"
         gcloud components list
         sudo apt-get update && sudo apt-get --only-upgrade install google-cloud-sdk-app-engine-python-extras google-cloud-sdk-datalab google-cloud-sdk-config-connector google-cloud-sdk-harbourbridge google-cloud-sdk-app-engine-java google-cloud-sdk google-cloud-sdk-spanner-emulator google-cloud-sdk-package-go-module google-cloud-sdk-skaffold kubectl google-cloud-sdk-terraform-tools google-cloud-sdk-kubectl-oidc google-cloud-sdk-cbt google-cloud-sdk-log-streaming google-cloud-sdk-local-extract google-cloud-sdk-pubsub-emulator google-cloud-sdk-app-engine-grpc google-cloud-sdk-app-engine-python google-cloud-sdk-app-engine-go google-cloud-sdk-anthos-auth google-cloud-sdk-nomos google-cloud-sdk-cloud-run-proxy google-cloud-sdk-firestore-emulator google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk-cloud-build-local google-cloud-sdk-minikube google-cloud-sdk-kpt google-cloud-sdk-bigtable-emulator google-cloud-sdk-datastore-emulator
         gcloud components list

Thank you for your help @igorboskovic3 Unfortunately, it seems that this also does not fix the issues: https://gist.github.com/chingucoding/3fad3f720bc57b73a62d0ddce3f2a017

It's quite strange that the gcloud CLI lists the auth plugin, yet it claims it is not installed. Could it be that we should uninstall the gcloud CLI before we do the upgrade?

@chingucoding Our Bitbucket pipelines started failing & I saw exact same errors you have mentioned. Our Pipelines are using google/cloud-sdk:latest so the solution is to replace "google/cloud-sdk:latest" with "google/cloud-sdk:debian_component_based" & then use "gcloud components install gke-gcloud-auth-plugin". Errors will resolve as they did for our pipelines.

- gcloud auth activate-service-account bitbucket-deployment@.....
- gcloud components install gke-gcloud-auth-plugin
- gke-gcloud-auth-plugin --version
- export USE_GKE_GCLOUD_AUTH_PLUGIN=True
- gcloud container clusters get-credentials ....

I hate doing a "me too" post but this is affecting me as well, and is a blocker for us to be able to deploy into gke.

I have tried the workarounds listed here but they do not work.

I'm using "ubuntu-latest" on my runner(s) -- so that's 22.04

I've even tried removing the apt versions of gcloud and installing the binary tarball directly, into /opt/gcloud/google-cloud-sdk/bin, and adding that to GITHUB_PATH, but this also fails with the same error, "error: The gcp auth plugin has been removed. Please use the "gke-gcloud-auth-plugin" kubectl/client-go credential plugin instead."

There's something seriously wrong with the GHA ubuntu images as far as the GKE auth plugin goes.

One more followup: I tested changing the runner to use Ubutnu 18.04 -- and it works. The only step that needs to run is the installation of the gcloud auth plugin -- sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin

The 18.04 runner has gcloud v406 installed

The plugin installation step installs v412 of the plugin.

Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (412.0.0-0) ...
Setting up google-cloud-sdk-gke-gcloud-auth-plugin (412.0.0-0) ...

Something broke between gcloud 411 and 412 on ubuntu 20.04 and 22.04

facing the same problem and can not deploy anymore:

error: The gcp auth plugin has been removed.
Please use the "gke-gcloud-auth-plugin" kubectl/client-go credential plugin instead.
See https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke for further details

despite the gke-gcloud-auth-plugin is already installed

gke-gcloud-auth-plugin --version                                                                                    
Kubernetes v1.25.2-alpha+ae91c1fc0c443c464a4c878ffa2a4544483c6d1f

any updates about this issue?

I find that despite the plugin being installed, kubectl will not try and use it until you set the environment variable USE_GKE_GCLOUD_AUTH_PLUGIN=True

I find that despite the plugin being installed, kubectl will not try and use it until you set the environment variable USE_GKE_GCLOUD_AUTH_PLUGIN=True

I've exported USE_GKE_GCLOUD_AUTH_PLUGIN="True" but problem is the same.

I think that I can understand the error need to : export USE_GKE_GCLOUD_AUTH_PLUGIN=True then again run : gcloud container clusters get-credentials $CLUSTER by this way the kubeconfig entry generated for the cluster the order is very important

I just ran into this before. The fix is to also install kubectl, replacing/upgrading the existing version on the runners I guess?

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt update
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin kubectl
export USE_GKE_GCLOUD_AUTH_PLUGIN=True
gcloud container clusters get-credentials ${{ inputs.cluster }} --region ${{ inputs.zone }}

I also use helm which doesn't have this problem in the same pipeline, just kubectl direct commands. Which is a weird interaction.

I can confirm that @acook-vhs's, @oladhari's and @dantheperson's solution work. The following snippet seems to work on Ubuntu 22.04:

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt update
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin kubectl
export USE_GKE_GCLOUD_AUTH_PLUGIN=True

gcloud auth activate-service-account azure-devops-service-account@YOUR_GOOGLE_CLOUD_PROJECT_NAME.iam.gserviceaccount.com \
          --key-file <(echo $(GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON)  | base64 -d)
(echo $(GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON) | base64 -d) > $HOME/.config/gcloud/application_default_credentials.json
gcloud container clusters get-credentials --zone "europe-west3" "YOUR_KUBERNETES_CLUSTER_NAME" --project=YOUR_GOOGLE_CLOUD_PROJECT_NAME
kubectl get pods

I can also confirm this solution.

Just ensure that USE_GKE_GCLOUD_AUTH_PLUGIN=True is exported to the shell env before running kubectl commands. Export that env var as the very first thing before any other commands.

I was able to get this to work on ubuntu-latest runner (which is 22.04 currently)

Thanks everyone for contribution.

I can confirm that @acook-vhs's, @oladhari's and @dantheperson's solution work. The following snippet seems to work on Ubuntu 22.04:

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt update
sudo apt-get install google-cloud-sdk-gke-gcloud-auth-plugin kubectl
export USE_GKE_GCLOUD_AUTH_PLUGIN=True

gcloud auth activate-service-account azure-devops-service-account@YOUR_GOOGLE_CLOUD_PROJECT_NAME.iam.gserviceaccount.com \
          --key-file <(echo $(GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON)  | base64 -d)
(echo $(GOOGLE_CLOUD_SERVICE_ACCOUNT_JSON) | base64 -d) > $HOME/.config/gcloud/application_default_credentials.json
gcloud container clusters get-credentials --zone "europe-west3" "YOUR_KUBERNETES_CLUSTER_NAME" --project=YOUR_GOOGLE_CLOUD_PROJECT_NAME
kubctl get pods

kubctl get pods

This one has a typo. It should be kubectl get pods

Thanks for the help @chingucoding !

@igorboskovic3

sudo gcloud container clusters get-credentials --zone "europe-west3" "YOUR_KUBERNETES_CLUSTER_NAME" --project=YOUR_GOOGLE_CLOUD_PROJECT_NAME

and

gcloud container clusters get-credentials --zone "europe-west3" "YOUR_KUBERNETES_CLUSTER_NAME" --project=YOUR_GOOGLE_CLOUD_PROJECT_NAME

Are resolving the same situation in 2 different ways.

sudo one isn't using the new GKE gcloud plugin even if you export USE_GKE_GCLOUD_AUTH_PLUGIN=True.
without sudo it works

I would like to clarify for people coming here from google - the command you want ahead of any gcloud commands is export USE_GKE_GCLOUD_AUTH_PLUGIN=True. It wasn't clear from above answers that the export is used IMO

The solution that worked for me is : export USE_GKE_GCLOUD_AUTH_PLUGIN=True gcloud components install gke-gcloud-auth-plugin Do follow this order.

Thanks for your inputs @tnovau and @hugadams, I've updated my comment and the issue accordingly to also show the solution right at the top of the issue and indicate that the export is important.

I have a bit (I think) different issue. I was able to run command gcloud container clusters get-credentials --zone "europe-west3" "YOUR_KUBERNETES_CLUSTER_NAME" --project=YOUR_GOOGLE_CLOUD_PROJECT_NAME and now it shows correctly the name of the cluster kubectl config current-context but when I try to execute other commands such as kubectl get pod or kubectl cluster-info I am getting: E0709 10:17:27.723595 34196 memcache.go:265] couldn't get current server API group list: Get "https://34.x.x.x/api?timeout=32s": getting credentials : exec: executable gke-gcloud-auth-plugin not found

It looks like you are trying to use a client-go credential plugin that is not installed.

To learn more about this feature, consult the documentation available at: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

In fact I do not see on the list the applicable package: Installed │ BigQuery Command Line Tool │ bq │ 1.6 MiB │ │ Installed │ Cloud Storage Command Line Tool │ gsutil │ 11.3 MiB │ │ Installed │ Google Cloud CLI Core Libraries │ core │ 20.9 MiB │ │ Installed │ kubectl │ kubectl │ < 1 MiB

But even I am not able to install:

gcloud components install gke-gcloud-auth-plugin
ERROR: (gcloud.components.install) The following components are unknown [gke-gcloud-auth-plugin].

According to the: https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

Also, when I run in GCP cluster cloud shell terminal the command kubectl gets nodes I am getting the output, but the same command on my local desktop finished with the same message as for kubectl get pod

I faced the same error. Installing the plug in resolved the issue.

$ gcloud components install gke-gcloud-auth-plugin

actions / runner-images