ApplicationSet updates all applications in the set when only one has changes

[slyshovsv]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

We have many ApplicationSets in use for our deployments and noticed a weird behaviour with the sync process. We are mostly using monorepos for different projects, so one repo/branch usually has 5-10-15 apps covered by 1 ApplicationSet.

When a change is pushed to at least one of the apps, Argo sees it and syncs the changed application to the latest commit/revision. Then, it also syncs the rest of the applications in the set to the latest commit (which is expected since they are tied to the same branch in the same git repo). However, during this sync, it updates all the applications with a newer replica set essentially causing a pod recreation in all deployments in all applications. The new pods don't have any changes and match the source config.

Testing with different combination of sync options like PrunePropagationPolicy, ApplyOutOfSyncOnly=true or automated.prune=true does not help - the result is the same.

I've found something related here https://github.com/argoproj/argo-cd/issues/19726, though, unsure if it is the cause.

To Reproduce

• Deploy an ApplicationSet with the same branch source for multiple apps. Here is one of the AppSets we are using:

  apiVersion: argoproj.io/v1alpha1
  kind: ApplicationSet
  metadata:
  name: AppSet1
  namespace: argocd
  spec:
  generators:
  - list:
    elements:
      - name: app1
        cluster_name: CLUSTER
        path: PATH1
        namespace: NS1
      - name: app2
        cluster_name: CLUSTER
        path: PATH2
        namespace: NS2
        ....
      - name: app3
        cluster_name: CLUSTER
        path: PATH3
        namespace: NS3
      - name: app4
        cluster_name: CLUSTER
        path: PATH4
        namespace: NS4
  template:
  metadata:
    name: '{{name}}'
    namespace: argocd
  spec:
    project: PROJECT
    source:
      helm:
        valueFiles:
          - values.yaml
      path: '{{path}}'
      repoURL: https://github.com/repo.git
      targetRevision: BRANCH
    destination:
      name: '{{cluster_name}}'
      namespace: '{{namespace}}'
    syncPolicy:
      automated:
        prune: true
      syncOptions:
        - PruneLast=true
        - ApplyOutOfSyncOnly=true

• Make an update to a single application - update its values file or template, etc.
• Wait for ArgoCD to start reconciling applications.

Expected behavior

Only the application/applications affected by the commit should be updated, the rest should be synced to the latest commit without actual resource updates.

Version

argocd: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T11:57:48Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: linux/amd64

Logs

Looking at the logs - it seems like the controller treats all applications as changed

time="2024-08-30T15:58:33Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 project=custom
time="2024-08-30T15:58:33Z" level=info msg="Comparing app state (cluster: https://10.40.196.2, namespace: deploy1)" application=argocd/app1
time="2024-08-30T15:58:33Z" level=info msg="GetRepoObjs stats" application=argocd/app1 build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=8 unmarshal_ms=7 version_ms=0
time="2024-08-30T15:58:34Z" level=info msg=Syncing application=argocd/app1 skipHooks=false started=false syncId=00151-JcrRl
time="2024-08-30T15:58:34Z" level=info msg="Tasks (dry-run)" application=argocd/app1 syncId=00151-JcrRl tasks="[Sync/0 resource apps/Deployment:deploy1/app1 obj->obj (,,)]"
time="2024-08-30T15:58:34Z" level=info msg="Applying resource Deployment/app1 in cluster: https://10.40.196.2, namespace: deploy1" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
time="2024-08-30T15:58:34Z" level=info msg="Skipping auto-sync: another operation is in progress" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 project=custom
time="2024-08-30T15:58:34Z" level=info msg="No status changes. Skipping patch" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 project=custom
time="2024-08-30T15:58:34Z" level=info msg="Reconciliation completed" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 comparison-level=1 dedup_ms=0 dest-name=dest_cluster dest-namespace=deploy1 dest-server="https://10.40.196.2" diff_ms=21 git_ms=8 health_ms=0 live_ms=8 patch_ms=0 project=custom setop_ms=0 settings_ms=0 sync_ms=0 time_ms=49
time="2024-08-30T15:58:34Z" level=info msg="Updating operation state. phase: Running -> Running, message: '' -> 'one or more tasks are running'" application=argocd/app1 syncId=00151-JcrRl
time="2024-08-30T15:58:34Z" level=info msg="Applying resource Deployment/app1 in cluster: https://10.40.196.2, namespace: deploy1" dry-run=none manager=argocd-controller serverSideApply=false serverSideDiff=false
time="2024-08-30T15:58:34Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'deployment.apps/app1 configured'" application=argocd/app1 kind=Deployment name=app1 namespace=deploy1 phase=Sync syncId=00151-JcrRl
time="2024-08-30T15:58:34Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: 'one or more tasks are running' -> 'successfully synced (all tasks run)'" application=argocd/app1 syncId=00151-JcrRl
time="2024-08-30T15:58:34Z" level=info msg="sync/terminate complete" application=argocd/app1 duration=257.856338ms syncId=00151-JcrRl
time="2024-08-30T15:58:34Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 project=custom
time="2024-08-30T15:58:34Z" level=info msg="Comparing app state (cluster: https://10.40.196.2, namespace: deploy1)" application=argocd/app1
time="2024-08-30T15:58:34Z" level=info msg="GetRepoObjs stats" application=argocd/app1 build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=8 unmarshal_ms=8 version_ms=0
time="2024-08-30T15:58:34Z" level=info msg="updated 'argocd/app1' operation (phase: Succeeded)" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 project=custom
time="2024-08-30T15:58:34Z" level=info msg="Sync operation to ef9f72db3c1e15d9387d5e1459d62498f1c118aa succeeded" application=app1 dest-namespace=deploy1 dest-server="https://10.40.196.2" reason=OperationCompleted type=Normal
time="2024-08-30T15:58:34Z" level=info msg="Skipping auto-sync: another operation is in progress" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 project=custom
time="2024-08-30T15:58:34Z" level=info msg="Updated sync status: OutOfSync -> Synced" application=app1 dest-namespace=deploy1 dest-server="https://10.40.196.2" reason=ResourceUpdated type=Normal
time="2024-08-30T15:58:34Z" level=info msg="Updated health status: Healthy -> Progressing" application=app1 dest-namespace=deploy1 dest-server="https://10.40.196.2" reason=ResourceUpdated type=Normal
time="2024-08-30T15:58:34Z" level=info msg="Update successful" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 project=custom
time="2024-08-30T15:58:34Z" level=info msg="Reconciliation completed" app-namespace=argocd app-qualified-name=argocd/app1 application=app1 comparison-level=1 dedup_ms=0 dest-name=dest_cluster dest-namespace=deploy1 dest-server="https://10.40.196.2" diff_ms=18 git_ms=9 health_ms=0 live_ms=11 patch_ms=20 project=custom setop_ms=0 settings_ms=0 sync_ms=0 time_ms=85

[agaudreault]

Hey @slyshovsv, you mention

However, during this sync, it updates all the applications with a newer replica set essentially causing a pod recreation in all deployments in all applications.

Argo CD is not responsible to manage Kubernetes resources or Pod lifecycle. It only deploy what is configured in your manifest. If the "hydration" of your manifest produce a different results, for instance, using random generators in Helm charts, then it will cause Kubernetes to update your resources. This usually is an anti-pattern, but it is a problem in what you sync, not with Argo CD.

https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#random-data is one example.

[slyshovsv]

@agaudreault Well, I had a thought about some label/annotation etc causing that, but the thing is, we have monorepos for more than just one appset and at still happens at the AppSet level. Like, we got a repo with charts for dev appset with 4 apps and tst appset with 4 apps living on the same branch. As you might have guessed, the only difference is the values file content.

When you update one app in the dev appset - all apps in this asset get refreshed. But the tst appset just quietly syncs to the same commit without updates. That's why I decided to dig into the appset controller.

[dennispan]

Ran into the same. We use applicationset to manage multiple applications. After we enable automated sync for the apps (via AppSet template), whenever one of the app updates, it ended up auto-syncing all apps in the AppSet

Worth noting that our apps are multi-source apps

[crenshaw-dev]

The ApplicationSet controller doesn't sync anything. It just applies Application objects to the Kubernetes API. The Application controller is then responsible for reconciling those Applications, which may or may not involve syncing them. The Application controller reconciles Applications completely independently of the ApplicationSet objects - it has no knowledge the ApplicationSet's state or existence.

@dennispan if you add an automated sync policy to your ApplicationSet's spec.template, then that policy will be applied to all Applications created by the ApplicationSet. If you need to conditionally add a sync policy to a subset of Applications, use templatePatch.

[dennispan]

@crenshaw-dev Thanks for the explanation. Adding automated sync policy to AppplicationSet adds it to all apps is what we want. That's not the issue.

The issue is that when one of the app updates, it ends up syncing all apps in the appset. What I would expect is it should only sync the updated app

[rumstead]

If an application controlled by the applicationset changes, the applicationset will check the fields "owned" by the applicationset. If the owned fields changed, it will reprocess all the applications the applicationset manages. However, that should result in a no-op by the applicationset controller since it should generate the same application.

Does something cause a drift between the generated applications from the applicationset?

[dennispan]

@rumstead We only changed the values.yaml file. Imagine we updated replica of a deployment from 1 to 2. I imagine that shouldn't be something "owned" by applicationset?

[agaudreault]

So I think there's some confusion on what syncing all the apps in the appset means.

If you update a value file and you push a new commit, all the applications that are syncing the repository and targetRevision will sync the new commit.

You can check https://argo-cd.readthedocs.io/en/release-2.0/operator-manual/high_availability/#webhook-and-manifest-paths-annotation to only sync when changes relevant to the application are detected.

If that's the behavior you experience, then this is totally unrelated to the use of appset.

[dennispan]

Got it. I tried webhook + manifest path annotation yesterday. It didn't work for us. It still ended up updating all apps.

In the manifest paths annotation doc, I didn't see examples with multi-source applications. And we use multi-source. I wonder if it does not support for multi-source app yet?

[bogatuadrian]

Update: The problem was on my end; I used a map in the helm template which was rendering in a different order for each helm template, leading ArgoCD to detect changes, and therefore sync the app(s).

~We also started to see a similar behavior recently. We recently updated to ArgoCD 2.11.7, so that might be the cause.~

~The issue is very annoying as we have post-sync hooks that trigger even though no changes have been made to any of the applications.~

~What's also worth mentioning is that ANY repo update (even if it doesn't affect any ArgoCD application) will trigger a re-sync.~

[rumstead]

What version did you upgrade from? When an application syncs that doesn’t have any changes what does the sync status say? Are any resources changed?

[bogatuadrian]

Sorry, I forgot to update my message from yesterday. I investigated the issue and it was a silly mistake on my part: I used a map in a helm template that was rendering in a different order each time, so ArgoCD always detected changes.