AppOfApp generated from AppSet has Applications which are flapping in between "Sync" and "OutOfSync"

[esn89]

Also posted to Slack here: https://cloud-native.slack.com/archives/C01TSERG0KZ/p1706035288683999 but no satisfactory answer

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Application from AppOfApp keeps flapping between synced and out of sync.

To Reproduce I have an AppSet which generates an AppOfApp for each cluster. The cluster secret looks like this:

---
apiVersion: v1
kind: Secret
metadata:
  labels:
    argocd.argoproj.io/secret-type: cluster
    env: dev
    region: us-central1
    branch: develop
    regionSize: small
  name: us-cluster-secret
  namespace: argocd
type: Opaque
stringData:
  name: us-cluster
  server: https://$IP
  config: |
    {
      "execProviderConfig": {
        "command": "argocd-k8s-auth",
        "args": ["gcp"],
        "apiVersion": "client.authentication.k8s.io/v1beta1"
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": ""
      }
    }

I have an AppSet which generates (along with progressive syncs enabled, the AppOfApps)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.region }}-data-management-svc'
  namespace: argocd
spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  goTemplate: true
  generators:
  - git:
      repoURL: https://code.com/cluster-secrets.git
      revision: master
      files:
      - path: "charts/{{ .Values.env }}/templates/main/*.yaml"
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: regionSize
              operator: In
              values:
                - small
        - matchExpressions:
            - key: regionSize
              operator: In
              values:
                - medium
  template:
    metadata:
      name: '{{`{{ .metadata.labels.project }}`}}-data-management-svc'
      labels:
        regionSize: '{{`{{ .metadata.labels.regionSize }}`}}'
    spec:
      ignoreDifferences:
      - group: "apps"
        kind: "Deployment"
        jsonPointers:
        - /spec/replicas
      destination:
        namespace: argocd
        server: 'https://kubernetes.default.svc'
      project: {{ .Values.env }}
      source:
        path: services/app-of-apps
        repoURL: https://code.com/appofapps.git
        targetRevision: {{ .Values.branch }}
        helm:
          parameters:
 <SKIPPED TO KEEP THINGS SHORT>
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          - RespectIgnoreDifferences=true

The AppOfApp has a bunch of Apps, each with a sync wave from 0-9 associated to it.

What's going on is that, only App with Sync Wave 8 is flapping between synced and unsynced. These are the changes it applies and reverts every minute:

screenshot

screenshot2

Expected behavior

I expect this diff to be applied once and not get reverted.

Screenshots

Version

{
    "Version": "v2.9.3+6eba5be",
    "BuildDate": "2023-12-01T23:05:50Z",
    "GitCommit": "6eba5be864b7e031871ed7698f5233336dfe75c7",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.13.2+g2a2fb3b",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

Logs

Paste any relevant application logs here.

[esn89]

Interestingly enough if I delete the AppOfApps, the same application that was flapping is the only one that does NOT get deleted:

screenshot

[esn89]

My application, which is part of an AppOfApps, is flapping between OutOfSync and Synced. Once it is synced by the Autosync, it goes back to OutOfSync, and I think I found the issue in the logs:

time="2024-01-24T19:33:43Z" level=info msg="Updating resource result, status: 'Synced' -> 'Synced', phase 'Running' -> 'Succeeded', message 'application.argoproj.io/cluster-us-provision-backend-server configured. Warning: resource applications/cluster-us-provision-backend-server is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by  apply.  apply should only be used on resources created declaratively by either  create --save-config or  apply. The missing annotation will be patched automatically.' -> 'application.argoproj.io/cluster-us-provision-backend-server configured. Warning: resource applications/cluster-us-provision-backend-server is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by  apply.  apply should only be used on resources created declaratively by either  create --save-config or  apply. The missing annotation will be patched automatically.'" application=argocd/cluster-us-data-management-svc kind=Application name=cluster-us-provision-backend-server namespace=argocd phase=Sync syncId=12237-gmzyQ

I compared this, with another App within the same AppOfApp, and they indeed have a "last-applied-configuration" on their manifest. However, if ArgoCD itself has synced it, then why isn't this annotation staying permanently in my Application's (cluster-us-provision-backend-server)'s manifest?

[esn89]

Issue closed. Turns out this resource was duplicated and another ApplicationSet was fighting to change it.