App of apps regularly gets out of sync when some apps have explictly specified empty sync policy

[andrii-korotkov-verkada]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Some applications managed by the app of apps explicitly specify syncPolicy: {}, sometimes in kustomize overlays for some shards, so just removing it is not an option. Apparently empty sync policy gets deleted somehow and ArgoCD thinks that app is out of sync since empty sync policy needs to be added. Server side diff is enabled, but not server side apply.

To Reproduce

Create an app of apps managing an application with syncPolicy: {}. Observe the behavior of app of apps for some time. Eventually when the field is deleted and refresh happens, it'd show up as out of sync and would try to update the app.

Expected behavior

App doesn't consider empty sync policy as being out of sync.

Screenshots

Version

Custom build from master around 2024/07/07 plus https://github.com/argoproj/argo-cd/pull/18972, https://github.com/argoproj/argo-cd/pull/18694, https://github.com/argoproj/gitops-engine/pull/601, https://github.com/argoproj/gitops-engine/pull/603

Logs

[andrii-korotkov-verkada]

This may be a duplicate of https://github.com/argoproj/argo-cd/issues/18213, however PRs there weren't enough to fix this. The EKS has Kubernetes 1.28.

[andrii-korotkov-verkada]

One caveat is sometime we want fields with empty value, e.g. when automated sync is enabled. So just equating empty value and absent value during a diff may not work as a fix.