Apps don't sync every are in stuck in refreshing

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes

https://argo-cd.readthedocs.io

Apache License 2.0

17.45k stars 5.3k forks source link

Apps don't sync every are in stuck in refreshing #18306

Open haooliveira84 opened 4 months ago

haooliveira84 commented 4 months ago

Checklist:

[x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[x] I've included steps to reproduce the bug.
[x] I've pasted the output of argocd version.

Describe the bug I've updated the ArgoCD from v2.9.5 to 2.11.0 with dynamicClusterDistribution: enabled and the ArgoCD Applications don't sync every are stuck in refreshing, I see the application controller does not have many logs.

To Reproduce 1 - Install Argocd with helm-chart with version: 6.9.3 2 - Edit the values.yaml and configure the controller to dynamicClusterDistribution: enabled

Expected behavior

The application controller balances the clusters with yours pods. Sync stuck resolved

haooliveira84 commented 3 months ago

tagging @ishitasequeira from https://github.com/argoproj/argo-cd/pull/15036

haooliveira84 commented 3 months ago

Anyone looking to this?

jenna-foghorn commented 3 months ago

Temp band-aid is to kick the argocd-application-controller via kubectl rollout restart statefulset argocd-application-controller -n argocd

oscrx commented 16 hours ago

I had this issue as well, I thought that it happened because the cluster ownership registration in the configmap was not being expired. I didn't look into it further then because I ran it on prod and people started complaining :)

Maybe the application controllers can use a Kubernetes lease for every cluster instead of a configmap to keep track of the ownership between application controllers.

oscrx commented 16 hours ago

Our configmap stays healthy after restarts, I'll keep it running for a while and see where it breaks. (3 instances currently)

❯ k get cm argocd-app-controller-shard-cm -o yaml | k neat
apiVersion: v1
data:
  shardControllerMapping: '[{"ShardNumber":0,"ControllerName":"argocd-application-controller-bb77ddc6-vlfhk","HeartbeatTime":"2024-09-18T07:06:37Z"},{"ShardNumber":1,"ControllerName":"argocd-application-controller-bb77ddc6-7fvrd","HeartbeatTime":"2024-09-18T07:06:36Z"},{"ShardNumber":2,"ControllerName":"argocd-application-controller-bb77ddc6-t8gpj","HeartbeatTime":"2024-09-18T07:06:35Z"}]'
kind: ConfigMap
metadata:
  name: argocd-app-controller-shard-cm
  namespace: argocd