ArgoCD reconciliation frequency for helm chart with dependencies much more than timeout.reconciliation

[dhirajsittu781]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: [https://bit.ly/argocd-faq.]
• [x] I've pasted the output of argocd version
• [x] I've tried using an older version of argocd: 2.1.7
• [x] I've included steps to reproduce the issue.

Describe the bug

I am deploying an application via ArgoCD with Helm chart as the source. The helm chart has a dependency chart for configmap only. When I remove this dependency, the ArgoCD app reconciles as per the value of timeout.reconciliation, but as soon as I add the dependency, the reconciliation happens much more frequently (every 4-5 seconds).

To Reproduce

1. Add a folder with helm chart templates (let's call this the main helm chart) in a git repository (or any other central repository).
2. Add a dependency in the above helm chart which is also stored in a central repository (ECR in my case).
3. Create an ArgoCD application with the above main helm chart as the source and an EKS cluster as the destination.
4. Observe the logs of the application controller pod searching for reconciliation operation for the above ArgoCD app.

Expected behavior

ArgoCD app should reconcile as per the value set in timeout.reconciliation.

Version

v2.7.4+a33baa3

[tribu]

Here is the sample log of the controller when there isa dependency chart specified in the main-charts Charts.yaml

time="2023-08-02T14:09:26Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:29Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:33Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:37Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:40Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:44Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:48Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:51Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:55Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:09:58Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:04Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:09Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:12Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:16Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:20Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:23Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:27Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:31Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app time="2023-08-02T14:10:34Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/main-app

This is causing ratelimit on our gitrepo and throwing 429 error: time="2023-08-02T14:09:55Z" level=info msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-08-02T14:09:54Z\",\"message\":\"rpc error: code = Unknown desc = unexpected client error: unexpected requesting \\"https://bitbucket.org/XXXXXXXXX/helm-services.git/info/refs?service=git-upload-pack\\" status code: 429\",\"type\":\"ComparisonError\"}]}}" application=argocd/main-app

[tribu]

[black-snow]

@tribu have you found a solution for this?

[faust64]

Hey

I've been digging around, as I have a very similar issue, when migrating some of my applications, from a custom Config Management Plugin, ton some Helm Charts.

From what I observed, the core of the problem somewhat involves ArgoCD Appliactions controller, comparing the .spec.sources and the .status.sync.comparedTo.source

In previous version of my app, I had some .spec.source.plugin definition (with some env values, the name of my plugin). That one's gone from my .spec.source today, I only have .spec.source.helm instead

Yet when I look at the status of those applications, the .comparedTo.source key includes both helm and old-plugin definition. it's been merged or something?

That's the trigger. application-controller gets crazy, telling us that .spec.source changed, while it didn't, on a loop, ... hitting github ratelimit.

One way to "fix" it, for now, would be to delete your Application, then re-create it / without any status, and let ArgoCD populates it back ... Should calm things down.