apps of apps - dependency (sync-wave) between apps not working

[vikas027]

Describe the bug

I am trying to make an app dependent on other apps. Just for sake of an example, I have taken two off the shelf charts - kubernetes-external-secrets and prometheus

NOTE: I have just taken these charts as an example, we have different charts (stored in a git repository) in my organization along with a few kustomized_helm charts (like kustomize-guestbook) which uses kustomize-helm plugin

To Reproduce I have tried to use sync-waves in vain, it does not look like it is coming into effect at all.

$ kubectl apply -f nonprod-apps.yaml

File nonprod-apps.yaml for calling other apps

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap-nonprod-apps
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: <my git repo>
    targetRevision: HEAD
    path: bootstrap/envs/nonprod
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

App files in the git repository

File 1 prometheus.yaml

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    chart: prometheus
    targetRevision: 13.3.2
    helm:
      releaseName: prometheus
      version: v3
      parameters:
      - name: 'alertmanager.enabled'
        value: 'false'
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

File 2 secrets-manager.yaml

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: secrets-manager
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  destination:
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://external-secrets.github.io/kubernetes-external-secrets
    chart: kubernetes-external-secrets
    targetRevision: 6.2.0
    helm:
      releaseName: secrets-manager
      version: v3
  syncPolicy:
    automated:
      selfHeal: true
      prune: true

Expected behavior

I want the status of the prometheus application to be healthy before argocd moves to secrets-manager application.

Screenshots

Version

❯ argocd version
argocd: v1.8.5+d0f8edf.dirty
  BuildDate: 2021-02-20T11:49:52Z
  GitCommit: d0f8edfec804c013d4fc535e8b9022eb47602617
  GitTreeState: dirty
  GoVersion: go1.15.8
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.8.4+28aea3d
  BuildDate: 2021-02-05T17:56:06Z
  GitCommit: 28aea3dfdede00443b52cc584814d80e8f896200
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.8.1 2020-07-16T00:58:46Z
  Helm Version: v3.4.1+gc4e7485
  Kubectl Version: v1.17.8
  Jsonnet Version: v0.17.0
❯

Related Issues / Questions

• https://github.com/argoproj/argo-cd/issues/3454
• https://github.com/argoproj/argo-cd/discussions/5316

[bravecobra]

I seem to have hit the same issue. Argo is not waiting for the helm chart release of App A to become healthy before helm releasing App B.

[aslafy-z]

Same issue, were my app is not an helm chart but points to a Git remote directory. It looks like it only happens sometimes, I was not able to reproduce this behavior consistently.

[jannfis]

For people hitting this issue, have you enabled health assessment for Argo CD Application resources - https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/1.7-1.8/#health-assessement-of-argoprojioapplication-crd-has-been-removed ?

[bravecobra]

I don't have health assessment enabled for Application and I'm still having the issue. Btw, I'm running 2.0.3

[aslafy-z]

@jannfis thank you for the tip. I've seen this page just landed in the docs in the last few days. It works like a charm. @bravecobra try enabling it!

[rogfut]

Had this same issue with argocd v2.0.3, the fix linked here to modify the config map worked, but I also had to enable autosync on my child apps.

Is sync wave supposed to work with autosync disabled? I have a 4 wave app of apps deployment and could not get sync waves to work without the configmap update and enabling autosync.

I can provide repro steps if need be.

argocd: v2.0.3+8d2b13d.dirty
  BuildDate: 2021-05-27T19:54:02Z
  GitCommit: 8d2b13d733e1dff7d1ad2c110ed31be4804406e2
  GitTreeState: dirty
  GoVersion: go1.16.4
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.0.4+0842d44
  BuildDate: 2021-06-23T01:27:53Z
  GitCommit: 0842d448107eb1397b251e63ec4d4bc1b4efdd6e
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.9.4 2021-02-09T19:22:10Z
  Helm Version: v3.5.1+g32c2223
  Kubectl Version: v0.20.4
  Jsonnet Version: v0.17.0

[netguino]

I am not sure either, I am looking at the docs, but I cannot for the life of me, get the child apps to autocreate without autosync.

[gimpiron]

Hey, any news on that issue? are you guys found some good workaround? I have tried to use neg / positive sync wave values but with no success - its not working on argocd/application on latest version.

[morningspace]

It looks I am having the same issue. I'm using ApplicationSet and created the top level application pointing to a set of ApplicationSet from UI, not sure if it introduces any difference. After the top level application created, the apps will be auto-generated after I registered a cluster (I'm using cluster generator), then I see all child applications are created automatically and some applications should have been created after others since I assigned different sync-wave values to them. However, I see they are synced and processing simultaneously. I've already added back the custom health check for Application resource.

[wmiller112]

I'm using ApplicationSet and created the top level application pointing to a set of ApplicationSet

Have an almost identical setup - top level Application pointing to directory with a few ApplicationSets. The resulting apps from the ApplicationSets have the appropriate wave annotations to install some of the apps before others, and they all show in the UI as part of the top level Application. This seems like it falls under the app of apps concept that should work based on other GH issue resolutions, yet they all install simultaneously. I do have the health check set up.

This isn't so much a problem for me on new cluster creation, or general install, because retries eventually get everything installed. It is on cluster deletion though, as operators are being deleted before they can operate on (clean up) their custom resources.

Bit more digging and I think this is what we're looking for https://github.com/argoproj/argo-cd/issues/9437, progressive rollouts for ApplicationSets

[ron1]

It seems the fix for the original issue here was to enable application health checking.

The recent discussion concerning the non-functional 'App of ApplicationSets' pattern may be addressed by https://github.com/argoproj/argo-cd/issues/9437.

[jwenz723]

The progressive rollout feature mentioned above is now available in experimental status: https://argo-cd.readthedocs.io/en/release-2.6/operator-manual/applicationset/Progressive-Rollouts/

Seems like this is the best way to solve this problem.

Though, I can think of situations where this is not appropriate. For example, imagine application1 is developed by team1 and application2 is developed by team2 and that application2 cannot start up until after application1 has started. In this scenario it doesn't make sense to deploy both application1 and application2 with a single applicationset since they are owned by different teams. I think in this situation the best thing to do is to have an initContainer in the pod spec for application2 that validates that application1 has started OR have code inside of application2 that does the check.

[fvogl]

Sync waves in 2.7.2 work only for the initial deployment of the apps and it's deletion. There the sequencing is properly working. But for any change in the apps, the sync does not wait for the apps in the earlier sync-waves to become healthy. And yes, I have configured the app health check.

[adam-harries]

Sync waves in 2.7.2 work only for the initial deployment of the apps and it's deletion. There the sequencing is properly working. But for any change in the apps, the sync does not wait for the apps in the earlier sync-waves to become healthy. And yes, I have configured the app health check.

@fvogl I'm seeing the same thing. Initial deployment works as expected but subsequent updates do not obey sync waves. I don't suppose you managed to find a workaround for this?

[fvogl]

@adam-harries unfortunately not, so actually for us this sync-wave feature is not usable, as we constantly evolve our microservice based platform and not install it once only.

[adam-harries]

@adam-harries unfortunately not, so actually for us this sync-wave feature is not usable, as we constantly evolve our microservice based platform and not install it once only.

I was afraid you'd say that :( Did you find an alternative approach to manage dependencies between app deployments?

[crenshaw-dev]

Several things to take away from this issue: 1) For sync waves to work among apps in an app-of-apps, you must enable the Application health check. 2) Sync waves in apps-of-apps order synchronization of the Application resources themselves, not (necessarily) the resources managed by those Applications. So if the manifests of the child Applications themselves have not changed, the parent app has nothing to sync and therefore no order is enforced. If you need something that is aware of child Apps' managed resources, you have a couple options: 1) hard-code the spec.source.targetRevision field in each Application and bump it whenever there's a change so the parent App has something to sync 2) try out ApplicationSets' Progressive Syncs, which does monitor the child apps' managed resources 3) Obviously, the docs for all the above need to be better