Waves are not respected when Server-side apply in use

[hau21um]

Checklist:

• [X] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [X] I've included steps to reproduce the bug.
• [X] I've pasted the output of argocd version.

Describe the bug When deploying 3 resouces in ArgoCD application, all with "default" apply and waves for ex -2,-1,0, all is deployed as expected (-2 - first, -1 second, 0 last). But when the last one with wave 0 is deployed with ServerSideApply=true, then is not applied last, but together with the first one (i.e. together with the -2 wave). To Reproduce Branch with waves only - https://github.com/hau21um/waves-ssa-demo/tree/waves Branch with waves+server-side-apply - https://github.com/hau21um/waves-ssa-demo/tree/waves_and_ssa

When is the server-side apply example run, the deployment from wave 0 nearly immediately receives

Warning  FailedMount  3s (x5 over 11s)  kubelet            MountVolume.SetUp failed for volume "my-cm" : configmap "my-cm" not found                                                                                                                                                                                                                      ```
As the config map is created in wave -1, so should be already created. If we run the example without server-side apply, then CM is created first and only after then the deployment from wave 0 is deployed.
There should be no difference in the waves ordering even when using server-side apply.

**Version**

```shell
argocd: v2.6.7+5bcd846.dirty
  BuildDate: 2023-03-23T17:23:10Z
  GitCommit: 5bcd846fa16e4b19d8f477de7da50ec0aef320e5
  GitTreeState: dirty
  GoVersion: go1.20.2
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.6.7+5bcd846.dirty
  BuildDate: 2023-03-23T17:23:10Z
  GitCommit: 5bcd846fa16e4b19d8f477de7da50ec0aef320e5
  GitTreeState: dirty
  GoVersion: go1.20.2
  Compiler: gc
  Platform: darwin/arm64
  Kustomize Version: v5.0.1 2023-03-14T01:20:22Z
  Helm Version: v3.11.2+g912ebc1
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

Paste any relevant application logs here.

[TheBritz]

I can confirm this behavior on v2.5.5+fc3eaec

Disabling ServerSideApply allows waves to work as expected, but with it enabled all resources deploy simultaneously as if no waves are being leveraged.

Problem for us is we have some large configmaps we were using ServerSideApply for so now we are caught between a rock and a hard place.

[hau21um]

Same story here, resource with ServerSideApply and wave="-2" is created with wave 0. Interestlingly when deleting this ArgoCD app, seems the waves are respected even for those resources with ServerSideApply enabled.

[ju-la-berger]

Hey there! This issue is very annoying for our setup as well. It seems this issue has not even been triaged and prioritized yet. :-(

[crenshaw-dev]

@ju-la-berger PRs are easier to prioritize than issues. 🙂

But tbh PRs are even difficult to grant priority right now. We need more long-term contributors moving up the maintainer ladder and getting Approver status. If your organization relies heavily on Argo CD, please ask for resources to become a regular contributor. That will help move fixes along.

[ju-la-berger]

@crenshaw-dev Thanks for your quick reply. I understand and will push this topic in my organization.

[yannrouillard]

I also faced the issue with v2.9.1.

After some tests using a local argocd instance, it seems to be still present in v2.9.3 but fixed in the master branch. One of the differences that could explain that is the update of gitops-engine dependency from v0.7.1-0.20230906152414-b0fffe419a0f to v0.7.1-0.20231102154024-c0c2dd1f6f48.

This version properly uses the server dry-run strategy when server-side apply is used starting from this commit https://github.com/argoproj/gitops-engine/commit/4a5648ee411b42f6601782ae6c1c50e0f724c9c5

It seems that with argocd <= 2.9.3, all resources are in fact created by mistake at the dry-run step when server-side is enabled, which would explains why everything is created at the first wave.

I suspect this is because dry-run client strategy is not really compatible with server-side apply mode, and instead of an error that should be raised, the resources are silently applied (but I wasn't able to confirm that)

The kubectl command refuses to apply such combination of settings:

❯ kubectl apply --server-side --dry-run=client -f foo.yaml
error: --dry-run=client doesn't work with --server-side (did you mean --dry-run=server instead?)

[TimWestmark]

I tested this with v2.10.0 and the sync-waves are working with server-side apply, now.

[chanakya-svt]

@TimWestmark are you using app-of-apps configuration and the sync waves are working with server side.

I defined app-of-app which creates 4 applications. I have sync-waves on each of the application. sync-waves are not respected in my case. I added the app health configMap and still not able to make it work.

[TimWestmark]

@chanakya-svt no, I do not have the app of apps pattern in place. Its one application containing multiple deployments