GKE Autopilot: argocd cluster add fails

[AndreasBergmeier6176]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

When I add a cluster to Argo (argocd cluster add gke_foo_europe-west1_haa) it fails with:

WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `gke_foo_europe-west1_haa` with full cluster level privileges. Do you want to continue [y/N]? y
FATA[0008] Failed to create service account "argocd-manager" in namespace "kube-system": serviceaccounts is forbidden: User "my@user.comn" cannot create resource "serviceaccounts" in API group "" in the namespace "kube-system": GKE Warden authz [denied by managed-namespaces-limitation]: the namespace "kube-system" is managed and the request's verb "create" is denied

This happens probably because Autopilot does not allow to modify kube-system.

To Reproduce

1. Create GCP Project foo
2. Create GKE Autopilot cluster in europe-west1 named haa
3. Add cluster to kubeconfig
4. argocd cluster add gke_foo_bar_haa

Expected behavior

It should somehow properly handle Autopilot security hardenings.

Version

argocd: v2.6.7+5bcd846
  BuildDate: 2023-03-23T15:24:49Z
  GitCommit: 5bcd846fa16e4b19d8f477de7da50ec0aef320e5
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64

Cluster is 1.25 btw.

[todaywasawesome]

Hmm, it's not a bug with Argo CD. It's an issue with GKE Autopilot. You need to find a way to make the service account for Argo CD to use.

[AndreasBergmeier6176]

Hmm, it's not a bug with Argo CD

Sorry I don't really understand. Are you saying that Argo is not supporting Autopilot? Is this documented?

[dtwilliamsWork]

I’m getting the same issue with an EKS cluster v1.25

[findheld-doug]

This problem appears to happen on GKE K8s clusters with Auto Pilot.

I was able to "add" the cluster by first creating an argo-cd namespace and then using the flag --system-namespace to the argocd cluster add command.

The cluster is configured now and running

I simply created a namespace called argo-cd

kubectl create namespace argo-cd

I then was able to add the cluster using the following command:

argocd cluster add \
  [my cluster name] \
  --system-namespace argo-cd

Hope this gets you unstuck

[minnie-jeong-otsk]

This may be a slightly different topic, but after adding the GKE autopilot cluster,

has anyone ever encountered error like Unable to create application: error while validating and normalizing app: error validating the repo: error getting k8s server version: Get "https://{clusterIP}/version?timeout=32s": x509: certificate signed by unknown authority , while trying to create the application on GKE autopilot cluster?

Anyone knows how to resolve this error?

[cyberslot]

I'm struggling with the same problem, but not on GKE Autopilot, just with private GKE (standard). Even after applying @findheld-doug's recipe the output is as follows:

argocd cluster add CLUSTER_NAME --system-namespace argo-cd
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `CLUSTER_NAME` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0001] ServiceAccount "argocd-manager" created in namespace "argo-cd" 
INFO[0002] ClusterRole "argocd-manager-role" updated    
INFO[0002] ClusterRoleBinding "argocd-manager-role-binding" updated 
INFO[0007] Created bearer token secret for ServiceAccount "argocd-manager" 
FATA[0038] rpc error: code = DeadlineExceeded desc = Get "https://{clusterIP}/version?timeout=32s": dial tcp {clusterIP}:443: i/o timeout

[soorena776]

Seems like findheld-doug's provided a fix for this issue. Should this be closed?

[daviinaa]

argocd cluster add <cluster name> --system-namespace argocd

this worked for me, first of all though, you might want to create the argocd name space and install argocd in the specified namespace in your autopliot cluster. should be able to do that with no issue.

[mahdikhashan]

not working with digital ocean cluster.