Switch to using gke-gcloud-auth-plugin to prevent build failures when the next version of k8s releases.

[novedevo]

Describe Request:

Switch to using gke-gcloud-auth-plugin to prevent build failures when the next version of k8s releases.

Examples:

Running the update-kubeconfig-with-credentials command results in the following bright red warning:

CRITICAL: ACTION REQUIRED: gke-gcloud-auth-plugin, which is needed for continued use of kubectl, was not found or is not executable. Install gke-gcloud-auth-plugin for use with kubectl by following https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

Supporting Documentation Links:

https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

[carmocca]

Bump! Our CI is flooded by a similar warning:

W0928 18:08:09.320122     748 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

[craiggunson]

Same here, wondering if I should add all the gcloud auth bits or wait?

[EricRibeiro]

This is a gcp-cli issue, so I'll move it there.

[prakash-fb]

Any update?

[Jaryt]

Any update?

Figured out the proper way to solve this, and working on getting the implementation in tonight. Should have a release ASAP.

[legopin]

So what is the solution now.

Today our CI begin to have full error instead of warning.

We were actually using the 1.3.0 version of circleci/gcp-gke@1.3.0 I was expecting that since our GKE server is still at 1.22 that the change wouldn't affect us, since the pinned version of the orb would use the older kubectl command

What is the advised fix? Use the newer version of circleci/gcp-gke@1.4.0?

error: The gcp auth plugin has been removed.
Please use the "gke-gcloud-auth-plugin" kubectl/client-go credential plugin instead.
See https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke for further details

[jordanderson]

Tonight we started getting the same error @legopin reports above while using circleci/gcp-gke@1.4.0.

[krainet]

Same here guys

[legopin]

I have confirmed that the installed latest version of kubectl is now 1.26.0

So looks like Google's definition of latest client stable recently changed to 1.26.0, which forced us to have the auth plugin

As a user of this orb I would expect that the plug in is installed along with kubectl

[ReccoD]

Same here

[philip-newzoo]

Any update?

Figured out the proper way to solve this, and working on getting the implementation in tonight. Should have a release ASAP.

Could we get a fix for this ASAP? We're running into the issue now and it's impacting our ability to deploy our releases.

[pokEarl]

Any update?

Figured out the proper way to solve this, and working on getting the implementation in tonight. Should have a release ASAP.

@Jaryt Hej we are also having some issues with this, if there is an easy way to fix this it would be great 🤪

[pokEarl]

Ok think I've made it work for our deploys at least 🎊 What I did:

1) Updated Gcp-cli version to 3.0.0 (changed /initialize to /setup and change credential keys to use _ instead of -)

2) Added a command to install it directly

• gcp-cli/install
• run: name: Installs new Gcloud Auth Plugin command: gcloud components install gke-gcloud-auth-plugin

3) Added the environment variable USE_GKE_GCLOUD_AUTH_PLUGIN=True

[rb46546741]

As a temporary fix, I'm currently trying to add a step k8s/install-kubectl with the parameter kubectl-version: v1.25.5 before running this orb with install-kubectl: false. I'll update this message if it works.

[EricRibeiro]

Hey folks

Thanks for sharing workarounds! Please stick to @pokEarl's suggestion while I work on a fix.

[willnevillain]

We ran into this too and it blocked our deploys; if it's helpful to others, here's our workaround in the meantime:

• add installation of a version tagged kubectl in an install_dependencies script we run at the beginning of our deploy job. Tagged to 1.25.4 before the gcp auth plugin was removed.
• remove the gke/install step (not required since gcloud was being installed earlier via a gcp-gcr/gcr-auth orb step)

Our script looks like this:

#!/bin/bash

set -e

# Create a user bin
_bin="$HOME/bin"
mkdir -v -p "$_bin"

# Install kubectl
kubectl_version=v1.25.4

wget https://dl.k8s.io/release/$kubectl_version/bin/linux/amd64/kubectl
chmod +x kubectl
mv kubectl $_bin

[neilpoulin]

the suggested fix doesn't work for me. I get the error shown below.

My job looks like this:

orbs:
  node: circleci/node@5.0.3
  slack: circleci/slack@4.4.4
  gcp-gcr: circleci/gcp-gcr@0.15.0
  gcp-gke: circleci/gcp-gke@1.4.0
  gcp-cli: circleci/gcp-cli@3.0.0

# Other configs/commands not shown

  deploy-to-dev:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - checkout
      - gcp-gke/install
      - gcp-cli/install # NEW JOB HERE
      - run: # NEW JOB HERE
          name: Installs new Gcloud Auth Plugin
          command: gcloud components install gke-gcloud-auth-plugin
      - run:
          name: 'Login to Google Cloud'
          command: |
            echo $GCLOUD_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
            gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
            gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
      - gcp-gke/rollout-image:
          cluster: XXX
          deployment: XXXX
          container: XXXX
          image: gcr.io/XXXXXX/XXXXX
          tag: $CIRCLE_SHA1
      - run: |
          curl -sL https://sentry.io/get-cli/ | bash
          export SENTRY_RELEASE=$CIRCLE_SHA1
          sentry-cli releases deploys $SENTRY_RELEASE new -e dev-stage

[EricRibeiro]

@neilpoulin, can you share the output of gcp-cli/install? The error message you received says that path.bash.inc does not exist. This file should've been downloaded from the CLI's install job.

Also, remove gcp-gke/install. You don't need it since you are installing the CLI via the gcp-cli orb. And the auth plugin can be installed via the CLI orb:

deploy-to-dev:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - checkout
      - gcp-cli/install:
           components: gke-gcloud-auth-plugin

[neilpoulin]

@EricRibeiro Thank you for the reply. I updated my job as you suggested, and now have a different error.

# Roll out image
/bin/bash: line 5: kubectl: command not found

Exited with code exit status 127

Any more help on this would be greatly appreciated, i'm trying get some critical changes out to production and am totally blocked.

Below is as much output as i could grab

I added the environment variable: USE_GKE_GCLOUD_AUTH_PLUGIN=True prior to running this build shown below.

# updated job snippet
  deploy-to-dev:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - checkout
      - gcp-cli/install:
          components: gke-gcloud-auth-plugin      
      - run:
          name: 'Login to Google Cloud'
          command: |
            echo $GCLOUD_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
            gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
            gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}   
      - gcp-gke/rollout-image:
          cluster: pluto-dev
          deployment: dev-pluto-web
          container: pluto-web
          image: gcr.io/pluto-staging-299000/pluto-web
          tag: $CIRCLE_SHA1
      - run: |
          curl -sL https://sentry.io/get-cli/ | bash
          export SENTRY_RELEASE=$CIRCLE_SHA1
          sentry-cli releases deploys $SENTRY_RELEASE new -e dev-stage

entire Job overview

Install latest gcloud CLI version full output

[EricRibeiro]

Thanks! It looks like all you need is kubectl now. You should be able to install it using the Kubernetes orb:

orbs:
  ...
  k8s: circleci/kubernetes@1.3.1

# Other configs/commands not shown

  deploy-to-dev:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - checkout
      - gcp-cli/install:
           components: gke-gcloud-auth-plugin
      - run:
          name: 'Login to Google Cloud'
          command: |
            echo $GCLOUD_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
            gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
            gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
      - k8s/install-kubectl
     ...

FYI, you can replace your entire Login to Google Cloud step with the CLI's initialize command; it takes care of the authentication for you.

[neilpoulin]

@EricRibeiro I got it working, but not with your latest suggestion. For completeness, here is the error thrown using your approach of using k8s/intstall-kubectl

Failed approach with k8s/install-kubectl

# Updated deploy job - errors out 
orbs:
  node: circleci/node@5.0.3
  slack: circleci/slack@4.4.4
  gcp-gcr: circleci/gcp-gcr@0.15.0
  gcp-gke: circleci/gcp-gke@1.4.0
  gcp-cli: circleci/gcp-cli@3.0.0
  k8s: circleci/kubernetes@1.3.1

# other commands/jobs not shown

  deploy-to-dev:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - checkout
      - gcp-cli/install:
          components: gke-gcloud-auth-plugin      
      - run:
          name: 'Login to Google Cloud'
          command: |
            echo $GCLOUD_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
            gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
            gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
      - k8s/install-kubectl
      - gcp-gke/rollout-image:
          cluster: pluto-dev
          deployment: dev-pluto-web
          container: pluto-web
          image: gcr.io/pluto-staging-299000/pluto-web
          tag: $CIRCLE_SHA1
      - run: |
          curl -sL https://sentry.io/get-cli/ | bash
          export SENTRY_RELEASE=$CIRCLE_SHA1
          sentry-cli releases deploys $SENTRY_RELEASE new -e dev-stage

Working config, install kubectl manually

  deploy-to-dev:
    machine:
      image: ubuntu-2004:202201-02
    steps:
      - checkout
      - gcp-cli/install:
          components: gke-gcloud-auth-plugin      
      - run:
          name: 'Login to Google Cloud'
          command: |
            echo $GCLOUD_SERVICE_KEY | gcloud auth activate-service-account --key-file=-
            gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
            gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
      - run:
          name: 'Install kubectl'
          command: |
            gcloud components install kubectl
      - gcp-gke/rollout-image:
          cluster: pluto-dev
          deployment: dev-pluto-web
          container: pluto-web
          image: gcr.io/pluto-staging-299000/pluto-web
          tag: $CIRCLE_SHA1

[EricRibeiro]

Awesome! Thanks for sharing, and I'm sorry that the orb is in this state. I'm working on a major release; hopefully, I can get it out there next week.

On another note, since you are installing kubectl as a component, you can use the CLI to do that for you. The components parameter accepts a space-separated list:

      - gcp-cli/install:
          components: gke-gcloud-auth-plugin kubectl

[corine-newzoo]

I can't get the temporary fix to work. We are using the gcp-gke/publish-and-rollout-image job instead of the gcp-gke/rollout-image command. I guess running the gcp-cli/install outside the gcp-gke/publish-and-rollout-image job does not work?

[phoenix2x]

For some mysterious reason, we could not make the gke-gcloud-auth-plugin work. The temporary workaround is to install v1.25 manually.

- k8s/install-kubectl:
          kubectl-version: v1.25.0

[EricRibeiro]

Hey folks 👋

I appreciate your patience! As I'm writing this, v2.0.0 is being published.

Please note that this is a major release. As such, breaking changes are expected, and you will need to update your config file to get it to work with v2.0.0. This was necessary due to the dire state the orb was in.

Check out the release notes to see what's changed and get an idea of where your config must be updated.

If you still see errors, please open issues, and I'll be happy to help!