search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
18.01k stars 5.49k forks source link

Sync waves complete when objects are still "Progressing" #4577

Open hintofbasil opened 4 years ago

hintofbasil commented 4 years ago

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

Describe the bug

When using the app of apps pattern with sync waves the sync waves completes when the application is marked as "Synced" but doesn't wait for the application to be "Healthy". If the application contains a deployment this results in the sync wave progressing when the new pods are not in a healthy state.

To Reproduce

Change the delay parameter in app_of_apps.yaml then apply the resource. This time the pods will be updated at the same time.

Expected behavior

On the second apply of the app of apps I would expect the first pod to reach a ready state before the second sync wave beings.

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v1.6.1+159674e.dirty
  BuildDate: 2020-07-21T14:27:57Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: dirty
  GoVersion: go1.14.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.6.2+3d1f37b
  BuildDate: 2020-07-31T23:45:07Z
  GitCommit: 3d1f37b0c53f4c75864dc7339e2831c6e6a947e0
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

Logs

Paste any relevant application logs here.
jessesuen commented 4 years ago

Because of numerous bugs like this with the app-of-apps pattern, we're actually removing the health status for Application resources in the v1.8 release -- we can't seem to get the sync waves with apps working correctly.

The long term solution is to introduce a AppSync CRD which can deal with coordinating syncing of multiple apps, correctly.

hintofbasil commented 4 years ago

Thanks for the heads up Jesse, we were investigating a design that relied on that status being accurate so very glad to know early.

Looking forward to reading the spec mentioned in https://github.com/argoproj/argo-cd/issues/1283.

ronaldour commented 4 years ago

@jessesuen Is there any known workaround to orchestrate the deployment order of applications in the meantime? We are facing the same issue, the sync-waves work perfectly on the initial deployment, but not when syncing the app after that. It seems that it just updates the Application resource but it doesn't wait for the actual resources of the application to be updated.

ronaldour commented 3 years ago

We created a custom health check for the argoproj.io/Application resource type and seems to be working so far. I'm not 100% sure it works all the time but it's worth trying as a workaround. 

argoproj.io/Application:
        health.lua: |
          hs = {}
          if obj.status ~= nil then
            if obj.status.sync.comparedTo.source.targetRevision ~= obj.spec.source.targetRevision then
              hs.status = "Progressing"
              hs.message = "Application Syncing"
              return hs
            else
              if obj.status.health.status == "Healthy" and obj.status.sync.status == "Synced" then
                hs.status = "Healthy"
                hs.message = "Application Ready"
                return hs
              end
            end
          end

          hs.status = "Progressing"
          hs.message = "Waiting for Application"
          return hs
rajivml commented 3 years ago

@ronaldour I have replaced the default application health check logic with yours and all apps got stuck in "Progressing" and "Waiting for Application" status , even though they are synced correctly. So, looks like there is something wrong with your code where as with the below code status is being reported correctly 

        hs = {}
        hs.status = "Healthy"
        hs.message = ""
        if obj.status ~= nil then
          if obj.status.health ~= nil then
            hs.status = obj.status.health.status
            hs.message = obj.status.health.message
          end
        end
        return hs

image

And if you notice, all of them are synced,

image