GKE cluster login example

achede22 commented 3 years ago

Description

Share GKE Exec plugin example so terraform can create cluster and continue configuring Kubernetes and HELM automatically

Potential Terraform Configuration

  kubernetes {
    host                   = var.cluster_endpoint
    # cluster_ca_certificate = base64decode(var.cluster_ca_cert)
    exec {
      api_version = "client.authentication.k8s.io/v1alpha1"
      args        = ["container", "clusters", "get-credentials", var.cluster_name, "--region", var.location, "--project", var.project]
      command     = "gcloud"
    }
  }
}

References

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

alexsomesan commented 3 years ago

@achede22 It's not entirely clear what you are proposing in this issue. Are you suggesting that we include this example in the documentation?

achede22 commented 3 years ago

Yes, please include this example in the documentation

saltyautomation commented 2 years ago

Can someone please add this example to the documentation.

sbocinec commented 2 years ago

Has any of commenters actually tried the proposed kubernetes provider exec config works?

It does not work for me (even changing the api_version to current v1beta or v1) and I don't think it can work as the command is designed to update your local kubeconfig. It does not return any token on the standard output usable by the exec.

https://cloud.google.com/sdk/gcloud/reference/container/clusters/get-credentials

gcloud container clusters get-credentials updates a kubeconfig file with appropriate credentials and endpoint information to point kubectl at a specific cluster in Google Kubernetes Engine.

The only way that I find working without relying on the google_client_config data source (that is omnipresent in GKE related documentations/articles) is to:

create kubeconfig using local_file provider, outputs from GKE cluster creation and use gcloud config config-helper dynamic block for the user auth

- name: ${cluster_identifier}
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

and then use config_path = /path/to/the/kubeconfig in the kubernetes/kubectl/helm provider configs.

This will ensure your code will not rely on the data source (that might not be refreshed when you need to run terraform commands with -refresh=false) and will keep the dynamism of the dynamically requested access token.

alnitech73 commented 1 year ago

It didn't work for me either, and I agree that is designed to generate kubeconfig not a token. Also using data.google_client_config was ok but only for the first terraform run and it stopped working after the token expired.

Then I looked into what gcloud container clusters get-credentials was adding to my kubeconfig and ended up using something similar to this. It works for me, when I deploy the cluster first time and in the subsequent terraform runs.

provider "kubernetes" {
  host  = var.cluster_endpoint
  cluster_ca_certificate = base64decode(var.cluster_ca_cert)
  exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      args        = []
      command     = "gke-gcloud-auth-plugin"
  }
}

fatmcgav commented 1 year ago

It didn't work for me either, and I agree that is designed to generate kubeconfig not a token. Also using data.google_client_config was ok but only for the first terraform run and it stopped working after the token expired.

Then I looked into what gcloud container clusters get-credentials was adding to my kubeconfig and ended up using something similar to this. It works for me, when I deploy the cluster first time and in the subsequent terraform runs.
provider "kubernetes" {
  host  = var.cluster_endpoint
  cluster_ca_certificate = base64decode(var.cluster_ca_cert)
  exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      args        = []
      command     = "gke-gcloud-auth-plugin"
  }
}

Just to add to this, it's also possible to leverage the gke-gcloud-auth-plugin whilst impersonating a GCP service account.

E.g.

provider "kubernetes" {
  host                   = "https://${module.gke.endpoint}"
  cluster_ca_certificate = base64decode(module.gke.ca_certificate)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    args        = ["--impersonate_service_account", var.impersonate_service_account]
    command     = "gke-gcloud-auth-plugin"
  }
}

x-yuri commented 5 months ago

In my opinion GKE authentication is best covered here. Because with the described way you don't need to run gcloud container clusters get-credentials before running terraform. Also it shows how to use the exec plugin with GKE in case you're curious.

The ways mentioned in the hashicorp/kubernetes documentation:

the implicit way is when you're running inside a cluster, which I bet is a really unusual case
the 2 ways where you need to reference files in ~/.kube is what I'd like to avoid, because generally I want to just run terraform, I don't want any prerequisite commands, and I guess I need to involve myself with contexts in case I have more than 1 cluster
the exec plugin way: no example for GKE, and it looked like it's not possible with GKE until I stumbled into the first link

At least that part (the first link) of the hashicorp/google documentation deserves a reference in hahicorp/kubernetes documentation. In case hashicorp/google is what is responsible for authentication.

hashicorp / terraform-provider-kubernetes