[ApplicationSet] Progressive sync blocks at a step even if all apps are healthy

Checklist:

[x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[x] I've included steps to reproduce the bug.
[x] I've pasted the output of argocd version.

Describe the bug

I am trying to use the Progressive Sync feature The doc mentions that All Applications in each group must become Healthy before the ApplicationSet controller will proceed to update the next group of Applications. But the rollout is blocked at step 1 whereas all applications in step 1 are Healthy (even if they are OutOfSync , which is desired as some custom resources cannot be deployed because of missing CRDs, but I configured these CRs with the argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true to not block the sync)

If I remove the resources that caused the "SyncFailed" status from the state and it unblocks the rolling...

Use Case:

My use case is :

multiple addons can be deployed to a cluster. I am using applicationset to generate the list of addons apps.
Most of the time, all the addons are deployed => no problem
But sometimes, to be more efficient in testing a specific feature, I want to deploy only some addons : let's call it the "minimal" addon list. (for example, in some cases, prometheus-operator will not be deployed)
some addons from the "minimal" list could include some CustomResources without the corresponding CRD being deployed (for example, an addon has a ServiceMonitor but prometheus-operator is not present, hence the ServiceMonitor CRD is missing)
To prevent sync to block in this case, all the CR are annotated with argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
Consequently, in the "minimal" case, all my apps can be Healthy but some can be OutOfSync because of the missing CRD that cause the CR to no being deployed.

And with this setting, I would expect that, as progressive sync step promotion is based on the Health status of the apps, it will finish its rollout successfully. But apparently, if the Sync is failed, the progressive rollout is blocked at the current step.

In the end, not sure if it is a bug or not but I would like to know if there is a way to get a successful progressive sync even if some resources are OutOfSync.

To Reproduce

Deploy an appset with progressive sync that generates apps which include one with a state containing custom resources without the corresponding CRD being present in the cluster (but including the SkipDryRunOnMissingResource=true option) The progressive sync sould be blocked at the step that includes this app

Expected behavior

I would expect all apps from the step 1 being Healthy a sufficient condition to trigger the step 2 as mentioned in the doc.

Slack discussion

https://cloud-native.slack.com/archives/C01U45M2SVB/p1725025472494469

Version

argocd: v2.12.3
  BuildDate: unknown
  GitCommit: v2.12.3
  GitTreeState: clean
  GitTag: v2.12.3
  GoVersion: go1.22.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.12.2+560953c
  BuildDate: 2024-08-23T03:30:19Z
  GitCommit: 560953c37b343c956f3a18f3db7d006e694c0dc4
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.4.2 2024-05-22T15:19:38Z
  Helm Version: v3.15.2+g1a500d5
  Kubectl Version: v0.29.6
  Jsonnet Version: v0.20.0

Logs

time="2024-08-30T13:24:40Z" level=info msg="ApplicationSet gtp-addons step list:" applicationset=argocd/gtp-addons
time="2024-08-30T13:24:40Z" level=info msg="step 1: [storage-class karpenter aws-ebs-csi-driver]" applicationset=argocd/gtp-addons
time="2024-08-30T13:24:40Z" level=info msg="step 2: [localdns]" applicationset=argocd/gtp-addons
time="2024-08-30T13:24:40Z" level=info msg="step 3: [draino]" applicationset=argocd/gtp-addons
time="2024-08-30T13:24:40Z" level=info msg="step 4: ]" applicationset=argocd/gtp-addons
time="2024-08-30T13:24:40Z" level=info msg="step 5: ]" applicationset=argocd/gtp-addons
time="2024-08-30T13:24:40Z" level=info msg="Application allowed to sync before maxUpdate?: map[aws-ebs-csi-driver:true draino:false karpenter:true localdns:false storage-class:true]" applicationset=argocd/gtp-addons
time="2024-08-30T13:24:40Z" level=info msg="triggering sync for application: karpenter, prune enabled: true" applicationset=argocd/gtp-addons
time="2024-08-30T13:24:40Z" level=info msg="triggering sync for application: aws-ebs-csi-driver, prune enabled: true" applicationset=argocd/gtp-addons
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="unchanged Application" app=argocd/draino applicationset=argocd/gtp-addons
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="unchanged Application" app=argocd/storage-class applicationset=argocd/gtp-addons
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="unchanged Application" app=argocd/karpenter applicationset=argocd/gtp-addons
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="patching application" app=argocd/aws-ebs-csi-driver applicationset=argocd/gtp-addons patch="map[operation:map[info:[map[name:Reason value:ApplicationSet RollingSync triggered a sync of this Application resource.]] initiatedBy:map[automated:true username:applicationset-controller] retry:map[] sync:map[prune:true]]]"
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=info msg="updated Application" app=argocd/aws-ebs-csi-driver applicationset=argocd/gtp-addons
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="unchanged Application" app=argocd/localdns applicationset=argocd/gtp-addons
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="received update event from owning an application" app=argocd/aws-ebs-csi-driver
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="requeue: false caused by application aws-ebs-csi-driver\n" app=argocd/aws-ebs-csi-driver requeue=false
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=info msg="end reconcile" applicationset=argocd/gtp-addons requeueAfter=3m0s
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="received update event from owning an application" app=argocd/aws-ebs-csi-driver
argocd-applicationset-controller-675fdd8b75-mz6zq argocd-applicationset-controller time="2024-08-30T13:47:07Z" level=debug msg="requeue: true caused by application aws-ebs-csi-driver\n" app=argocd/aws-ebs-csi-driver requeue=true

argoproj / argo-cd

[ApplicationSet] Progressive sync blocks at a step even if all apps are healthy #19771