Using Anthos Connect Gateway to authenticate to GKE cluster registered to a fleet

[rochana-atapattu]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Connect gateway provides an easy way to manage access to a GKE cluster registered to a fleet, specially ones without a public endpoint. We can use it to provide access to terraform resources in clusters without using VPN/Bastions etc.

New or Affected Resource(s)

• google_client_config data source?
• kubernetes provider
• helm provider

Potential Terraform Configuration

maybe a option to specify to use connect gateway when establishing identity.

[kartoch]

It is required to solve the connection problem to a fully private cluster (endpoint and nodes).

[rileykarson]

@SarahFrench to triage

[SarahFrench]

If I understand correctly, the request is to be able to access the kubeconfig returned by the gcloud container fleet memberships get-credentials command described here. A data source would make sense for that, depending on the API.

However I can't find the API endpoint that returns that kubeconfig information. I believe that gcloud container fleet memberships commands interact with the projects.locations.memberships resource in the API, but in the methods listed under that resource I can't find one that's equivalent to the get-credentials part of the gcloud command we want to mimic.

If I've overlooked anything please let me know, but for now I'll mark the issue as upstream

[kartoch]

For reference, the argocd-k8s-auth is used for authentication to a gke cluster via the fleet connect gateway based on a service account (with roles GKE Hub Admin and Kubernetes Engine Admin, this last one can be changed for a least privileged role).

The source code is here: https://github.com/argoproj/argo-cd/blob/master/cmd/argocd-k8s-auth/commands/gcp.go

[kartoch]

I've been able to authenticate using connect gateway:

• you need to have the connect-gateway URL (composed of the project id, the cluster region and the cluster name).
• you can then reuse the URL as host (so the connect gateway URL is a kubectl-compatible endpoint).

The relevant code:

# the private gke cluster
module "gke" {
  source = "terraform-google-modules/kubernetes-engine/google//modules/safer-cluster"
  enable_private_endpoint    = true
  ...
}

# the hub module makes private cluster accessible via connect gateway 
module "hub" {
  source       = "terraform-google-modules/kubernetes-engine/google//modules/fleet-membership"
  project_id   = var.project_id
  cluster_name = module.gke.name
  location     = module.gke.location
}

data "google_client_config" "default" {}

# to get the project id as number
data "google_project" "project" {
  project_id = var.project_id
}

# the URL to access the connect gateway
locals {
  connect_gateway = "https://connectgateway.googleapis.com/v1/projects/${data.google_project.project.number}/locations/global/gkeMemberships/${var.project_id}-${var.region}-${var.cluster_name}"
}

# the kubernetes provider using the connect gateway to access the private cluster
provider "kubernetes" {
  host                   = local.connect_gateway
  token                  = data.google_client_config.default.access_token
}

[ivanaguilario]

@kartoch is that connect_gateway URL documented somewhere? Can't find it in the GCP docs

[kartoch]

Not so much....

There are some basic examples in the following blog entries

• Connect Gateway and ArgoCD: Deploy to Distributed Kubernetes
• Building a Fleet of GKE clusters with ArgoCD (see also the GitHub source)

But these two example uses a previous connect gateway version (v1beta1) which does not seem to work now:

https://connectgateway.googleapis.com/v1beta1/projects/$PROJECT_NUMBER/locations/global/gkeMemberships/$CLUSTER_REGISTRATION_NAME

The format for v1 is the same and is working (found in this blog entry and the associated GitHub repository)

https://connectgateway.googleapis.com/v1/projects/$PROJECT_NUMBER/locations/global/gkeMemberships/$CLUSTER_REGISTRATION_NAME

With:

• $PROJECT_NUMBER, the value is the number and NOT the project ID.
• $CLUSTER_REGISTRATION_NAME you can get if with the command:

gcloud container fleet memberships list

[PaulSonOfLars]

For future travellers who might end up on this issue:

GKE clusters use: https://connectgateway.googleapis.com/v1beta1/projects/$PROJECT_NUMBER/locations/global/gkeMemberships/$MEMBERSHIP_ID Non-GKE clusters (eg, GKE-on-AWS): https://connectgateway.googleapis.com/v1beta1/projects/$PROJECT_NUMBER/locations/global/memberships/$MEMBERSHIP_ID

Note the difference between gkeMemberships and memberships for non-gcp clusters.