Argo CD hammering k8s api-server for cluster secrets

[nhavens]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Our Argo CD deploys both to the same cluster as well as multiple additional clusters. We have > 300 Application resources, many of which are generated by ApplicationSets. We see from the audit logs for our k8s api-server that an alarmingly high percentage of total requests, particularly those using the list verb come from the ApplicationSet controller (see Logs section below). This behavior was also present on Argo CD version 2.5.3.

To Reproduce

Deploy several hundred Applications to a single Argo CD instance, mostly generated by ApplicationSets.

Expected behavior

There should not be significant load places on the k8s api-server to fetch the Secrets required to reconcile these apps. As with many other components of Argo CD, some sort of caching mechanism should be put in place.

Version

argocd: v2.7.11+ec195ad
  BuildDate: 2023-08-07T19:41:50Z
  GitCommit: ec195adad84c61c6151d553b9fdce3c258b1325d
  GitTreeState: clean
  GoVersion: go1.19.11
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.7.11+ec195ad.dirty
  BuildDate: 2023-08-07T14:40:17Z
  GitCommit: ec195adad84c61c6151d553b9fdce3c258b1325d
  GitTreeState: dirty
  GoVersion: go1.19.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.0.1 2023-03-14T01:32:48Z
  Helm Version: v3.11.2+g912ebc1
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

Here's an example audit log from our k8s api-server.

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "8f804abf-1c70-404a-89a0-6b36ba89880b",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/argo-cd/secrets?labelSelector=argocd.argoproj.io%2Fsecret-type%3Dcluster",
  "verb": "list",
  "user": {
    "username": "system:serviceaccount:argo-cd:argocd-applicationset-controller",
    "uid": "9ac75bea-e4dd-427f-9fd7-5c8c05bbf5f1",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:argo-cd",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "10.16.156.39"
  ],
  "userAgent": "argocd-applicationset-controller/v0.0.0 (linux/amd64) kubernetes/$Format",
  "objectRef": {
    "resource": "secrets",
    "namespace": "argo-cd",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 200
  },
  "requestReceivedTimestamp": "2023-09-18T16:47:39.937436Z",
  "stageTimestamp": "2023-09-18T16:47:39.940412Z",
  "annotations": {
    "authentication.k8s.io/legacy-token": "system:serviceaccount:argo-cd:argocd-applicationset-controller",
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by RoleBinding \"argo-cd-applicationset-controller/argo-cd\" of Role \"argo-cd-applicationset-controller\" to ServiceAccount \"argocd-applicationset-controller/argo-cd\""
  }
}

[dberuben]

We're experiencing the same issue with numerous API calls. @nhavens , did you find a workaround?

{
    "Version": "v2.11.2+25f7504",
    "BuildDate": "2024-05-23T13:32:13Z",
    "GitCommit": "25f7504ecc198e7d7fdc055fdb83ae50eee5edd0",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.9",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.14.4+g81c902a",
    "KubectlVersion": "v0.26.11",
    "JsonnetVersion": "v0.20.0"
}