search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.59k stars 5.36k forks source link

manifest-generate-paths with autosync causes an undesirable refresh sync #18326

Closed zswanson closed 4 weeks ago

zswanson commented 4 months ago

Checklist:

Describe the bug

manifest-generate-paths behavior was adjusted in the 2.11.0 release to properly align with user expectations in mono-repos to control syncing of applications to specific paths. There is apparently some leftover logic in the app controller or newly introduced though, where upon detecting drift in a cluster resource for an app using manifest-generate-paths and autosync, that the controller does a 'refresh' sync to the HEAD commit of the repo (even though that may not related to the path of the managed application). If for some reason a cluster operator is attempting to make a manual change temporarily to an application (ie, during an incident) this has the effect of reverting the manual change, almost instantly. After that refresh sync, the cluster operator is able to apply the desired manual change again and then it will persist. We have explicitly set selfHeal to false in order to allow occasional imperative actions by our cluster operators, so this is undesirable behavior.

To Reproduce

  1. setup argocd 2.11.0 with github webhooks automation.
  2. Create an application with the annotation manifest-generate-paths: . and set spec.syncPolicy.automated to enabled, with selfHeal: false. The application source should be some sub-path in the repository like '/app1/` This will require github webhooks.
  3. Introduce a git change elsewhere in the repo. Should observe that the App does not sync, and its last sync state reflects an earlier commit in the git repo history.
  4. Manually change some aspect of the managed resources in the App; such as increasing a Deployment replicas. (Deployment changes illustrate this problem very well due to the 'generation' field) In the ArgoCD UI you will see a brief flurry of activity as the change is made and immediately rolled back. If you inspect the Deployment manifest now, the change is reverted and you'll see that the 'generation' increased by 2 (+1 for your change, and +1 again from Argo reverting it). The sync state and last sync state will now indicate they are pointing at the head commit of the repo.
  5. Make the same change again, this time it will persist and you'll see the application is out of sync.

Expected behavior

The app controller should not sync the App to the head commit, and the manual introduced change should persist on the first attempt.

Screenshots

Version

11:45:59 ❯ argocd version
argocd: v2.11.0+d3f33c0
  BuildDate: 2024-05-07T18:31:19Z
  GitCommit: d3f33c00197e7f1d16f2a73ce1aeced464b07175
  GitTreeState: clean
  GoVersion: go1.22.2
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.11.0+d3f33c0
  BuildDate: 2024-05-07T18:31:19Z
  GitCommit: d3f33c00197e7f1d16f2a73ce1aeced464b07175
  GitTreeState: clean
  GoVersion: go1.22.2
  Compiler: gc
  Platform: darwin/arm64
  Kustomize Version: v5.2.1 2023-10-19T20:11:23Z
  Helm Version: v3.13.2+g2a2fb3b
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

Logs

Paste any relevant application logs here.
agaudreault commented 3 months ago

Hey @zswanson, just to make sure I understand correctly, basically selfHeal=false is not respected when the application sync revision is not HEAD, and it works when the sync revision is HEAD?

Also, can you add the logs, mostly related to https://github.com/argoproj/argo-cd/blob/0f72c19e31481c38774a26e8147c1e3bfd5e0ecc/controller/appcontroller.go#L1815 and your application status before the kubernetes change, after the first automated sync and after the second kubernetes change.

I think the https://github.com/argoproj/argo-cd/blob/0f72c19e31481c38774a26e8147c1e3bfd5e0ecc/controller/appcontroller.go#L1944 method will return false when it should return true.

zswanson commented 3 months ago

Hey sorry I kept forgetting to respond here.

I would not say it's accurate to describe this as the self heal not being respected. Rather, it seems that if you are making a manual change to an application from the minor repo, and that application wad not already sync'd to HEAD from its last state that the controller first performs a sync to HEAD. That has the effect of undoing whatever change you'll manually tried to make. Them after that sync you can now make a manual change in-cluster.

I'll try this week to see if I can get some logs around it.

zswanson commented 3 months ago

current app state in argo before I make any changes

image 
07-01 13:37:58 time="2024-07-01T17:37:58Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/reloader
07-01 13:37:58 time="2024-07-01T17:37:58Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:37:58 time="2024-07-01T17:37:58Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=10 fields.level=2 git_ms=574 health_ms=0 live_ms=66 patch_ms=30 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=1629

via the CLI I scaled the deployment owned by the App to 3 replicas. (I've removed the 'normalized app spec' lines as they contained a lot of company internals I'd need to strip out anyways)

07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=9 unmarshal_ms=2 version_ms=6
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=error msg="DiffFromCache error: error getting managed resources for app reloader: cache: key is missing"
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Initiated automated sync to '65bda3a9a45e9d28ab938885d40a3602dcff4c9c'" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=OperationStarted spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Initiated automated sync to '65bda3a9a45e9d28ab938885d40a3602dcff4c9c'" application=argocd/reloader
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Updated sync status: Synced -> OutOfSync" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Updated health status: Healthy -> Progressing" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="updated 'argocd/reloader' operation (phase: Running)" appNamespace=argocd application=reloader project=repo.XXXXXXXXX
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:65bda3a9a45e9d28ab938885d40a3602dcff4c9c,Prune:true,DryRun:false,SyncStrategy:nil,Resources:[]SyncOperationResource{},Source:nil,Manifests:[],SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Sources:[]ApplicationSource{},Revisions:[],} { true} [] {-1 &Backoff{Duration:5s,Factor:*2,MaxDuration:10m,}}}" application=argocd/reloader
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=6 unmarshal_ms=1 version_ms=5
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=13 fields.level=1 git_ms=9 health_ms=0 live_ms=37 patch_ms=34 setop_ms=32 settings_ms=0 sync_ms=0 time_ms=329
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=1 version_ms=5
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg=Syncing application=argocd/reloader skipHooks=false started=false syncId=03053-squrJ
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Namespace already exists" application=argocd/reloader namespace=reloader syncId=03053-squrJ
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Tasks (dry-run)" application=argocd/reloader syncId=03053-squrJ tasks="[Sync/0 resource /ServiceAccount:reloader/reloader-reloader obj->obj (,,), Sync/0 resource rbac.authorization.k8s.io/ClusterRole:reloader/reloader-reloader-role obj->obj (,,), Sync/0 resource rbac.authorization.k8s.io/ClusterRoleBinding:reloader/reloader-reloader-role-binding obj->obj (,,), Sync/0 resource apps/Deployment:reloader/reloader-reloader obj->obj (,,)]"
07-01 13:46:55 time="2024-07-01T17:46:55Z" level=info msg="Applying resource ServiceAccount/reloader-reloader in cluster: https://172.18.32.1:443, namespace: reloader" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Skipping auto-sync: another operation is in progress" application=argocd/reloader
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=280 fields.level=1 git_ms=7 health_ms=94 live_ms=66 patch_ms=20 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=1721
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=33 unmarshal_ms=1 version_ms=31
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Applying resource ClusterRole/reloader-reloader-role in cluster: https://172.18.32.1:443, namespace: reloader" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Applying resource ClusterRoleBinding/reloader-reloader-role-binding in cluster: https://172.18.32.1:443, namespace: reloader" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Applying resource Deployment/reloader-reloader in cluster: https://172.18.32.1:443, namespace: reloader" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Updating operation state. phase: Running -> Running, message: '' -> 'one or more tasks are running'" application=argocd/reloader syncId=03053-squrJ
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Applying resource ServiceAccount/reloader-reloader in cluster: https://172.18.32.1:443, namespace: reloader" dry-run=none manager=argocd-controller serverSideApply=true serverSideDiff=false
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'serviceaccount/reloader-reloader serverside-applied'" application=argocd/reloader kind=ServiceAccount name=reloader-reloader namespace=reloader phase=Sync syncId=03053-squrJ
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Applying resource ClusterRole/reloader-reloader-role in cluster: https://172.18.32.1:443, namespace: reloader" dry-run=none manager=argocd-controller serverSideApply=true serverSideDiff=false
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'clusterrole.rbac.authorization.k8s.io/reloader-reloader-role reconciled. clusterrole.rbac.authorization.k8s.io/reloader-reloader-role serverside-applied'" application=argocd/reloader kind=ClusterRole name=reloader-reloader-role namespace=reloader phase=Sync syncId=03053-squrJ
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Applying resource ClusterRoleBinding/reloader-reloader-role-binding in cluster: https://172.18.32.1:443, namespace: reloader" dry-run=none manager=argocd-controller serverSideApply=true serverSideDiff=false
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding reconciled. clusterrolebinding.rbac.authorization.k8s.io/reloader-reloader-role-binding serverside-applied'" application=argocd/reloader kind=ClusterRoleBinding name=reloader-reloader-role-binding namespace=reloader phase=Sync syncId=03053-squrJ
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Applying resource Deployment/reloader-reloader in cluster: https://172.18.32.1:443, namespace: reloader" dry-run=none manager=argocd-controller serverSideApply=true serverSideDiff=false
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'deployment.apps/reloader-reloader serverside-applied'" application=argocd/reloader kind=Deployment name=reloader-reloader namespace=reloader phase=Sync syncId=03053-squrJ
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: 'one or more tasks are running' -> 'successfully synced (all tasks run)'" application=argocd/reloader syncId=03053-squrJ
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="sync/terminate complete" application=argocd/reloader duration=2.3129418s syncId=03053-squrJ
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="updated 'argocd/reloader' operation (phase: Succeeded)" appNamespace=argocd application=reloader project=repo.XXXXXXXXX
07-01 13:46:57 time="2024-07-01T17:46:57Z" level=info msg="Sync operation to 65bda3a9a45e9d28ab938885d40a3602dcff4c9c succeeded" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=OperationCompleted spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:46:58 time="2024-07-01T17:46:58Z" level=info msg="Skipping auto-sync: another operation is in progress" application=argocd/reloader
07-01 13:46:58 time="2024-07-01T17:46:58Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:46:58 time="2024-07-01T17:46:58Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=1 fields.level=1 git_ms=33 health_ms=0 live_ms=314 patch_ms=24 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=827
07-01 13:46:58 time="2024-07-01T17:46:58Z" level=info msg="Refreshing app status (controller refresh requested), level (2)" application=argocd/reloader
07-01 13:46:58 time="2024-07-01T17:46:58Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:46:58 time="2024-07-01T17:46:58Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=719 unmarshal_ms=320 version_ms=398
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/reloader
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Updated sync status: OutOfSync -> Synced" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Updated health status: Progressing -> Healthy" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=10 fields.level=2 git_ms=719 health_ms=0 live_ms=705 patch_ms=28 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=2174
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=1 version_ms=5
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Skipping auto-sync: application status is Synced" application=argocd/reloader
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=2 fields.level=1 git_ms=7 health_ms=0 live_ms=92 patch_ms=64 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=331
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=23 setop_ms=0 time_ms=176
07-01 13:47:00 time="2024-07-01T17:47:00Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=21 setop_ms=0 time_ms=181
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=19 setop_ms=0 time_ms=305
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=20 setop_ms=0 time_ms=181
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:01 time="2024-07-01T17:47:01Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=19 setop_ms=0 time_ms=170
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=20 setop_ms=0 time_ms=172
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=19 setop_ms=0 time_ms=179
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=25 setop_ms=0 time_ms=161
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:02 time="2024-07-01T17:47:02Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=19 setop_ms=0 time_ms=313
07-01 13:47:03 time="2024-07-01T17:47:03Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:03 time="2024-07-01T17:47:03Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:03 time="2024-07-01T17:47:03Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=19 setop_ms=0 time_ms=161
07-01 13:47:03 time="2024-07-01T17:47:03Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:47:03 time="2024-07-01T17:47:03Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:47:03 time="2024-07-01T17:47:03Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=19 setop_ms=0 time_ms=165

So now in the UI the sync state has changed to match HEAD. Weirdly there is no new replicaset, which it really should have created from changing the replica field via kubectl scale

image

Ran the command a second time - now the UI shows it as out of sync and 3 pods are present for the deploy. New logs:

07-01 13:53:29 time="2024-07-01T17:53:29Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:53:29 time="2024-07-01T17:53:29Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:53:29 time="2024-07-01T17:53:29Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=1 version_ms=5
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Skipping auto-sync: most recent sync already to 65bda3a9a45e9d28ab938885d40a3602dcff4c9c" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Updated sync status: Synced -> OutOfSync" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Updated health status: Healthy -> Progressing" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=9 fields.level=1 git_ms=7 health_ms=0 live_ms=37 patch_ms=47 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=297
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=8 unmarshal_ms=2 version_ms=5
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Skipping auto-sync: most recent sync already to 65bda3a9a45e9d28ab938885d40a3602dcff4c9c" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=9 fields.level=1 git_ms=8 health_ms=0 live_ms=59 patch_ms=20 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=347
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=1 version_ms=5
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Skipping auto-sync: most recent sync already to 65bda3a9a45e9d28ab938885d40a3602dcff4c9c" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=9 fields.level=1 git_ms=7 health_ms=0 live_ms=34 patch_ms=19 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=234
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:30 time="2024-07-01T17:53:30Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=20 setop_ms=0 time_ms=151
07-01 13:53:31 time="2024-07-01T17:53:31Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:53:31 time="2024-07-01T17:53:31Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:31 time="2024-07-01T17:53:31Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=43 setop_ms=0 time_ms=179
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=21 setop_ms=0 time_ms=157
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=19 setop_ms=0 time_ms=161
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:53:32 time="2024-07-01T17:53:32Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=1 version_ms=5
07-01 13:53:33 time="2024-07-01T17:53:33Z" level=info msg="Skipping auto-sync: most recent sync already to 65bda3a9a45e9d28ab938885d40a3602dcff4c9c" application=argocd/reloader
07-01 13:53:33 time="2024-07-01T17:53:33Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:33 time="2024-07-01T17:53:33Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=9 fields.level=1 git_ms=7 health_ms=0 live_ms=128 patch_ms=19 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=338
07-01 13:53:33 time="2024-07-01T17:53:33Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:53:33 time="2024-07-01T17:53:33Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:33 time="2024-07-01T17:53:33Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=24 setop_ms=0 time_ms=152
07-01 13:53:34 time="2024-07-01T17:53:34Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:53:34 time="2024-07-01T17:53:34Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:34 time="2024-07-01T17:53:34Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=20 setop_ms=0 time_ms=155
07-01 13:53:34 time="2024-07-01T17:53:34Z" level=info msg="Refreshing app status (controller refresh requested), level (0)" application=argocd/reloader
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="Reconciliation completed" application=argocd/reloader dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" fields.level=0 patch_ms=19 setop_ms=0 time_ms=163
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/reloader
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: reloader)" application=argocd/reloader
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="GetRepoObjs stats" application=argocd/reloader build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=7 unmarshal_ms=1 version_ms=5
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="Skipping auto-sync: most recent sync already to 65bda3a9a45e9d28ab938885d40a3602dcff4c9c" application=argocd/reloader
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="Updated health status: Progressing -> Healthy" application=reloader dest-namespace=reloader dest-server="https://kubernetes.default.svc" reason=ResourceUpdated spec="{&ApplicationSource{RepoURL:https://github.com/XXXXXXXXX/XXXXXXXXX,Path:clusters/XXXXXXXXX/reloader,TargetRevision:,Helm:nil,Kustomize:nil,Directory:nil,Plugin:&ApplicationSourcePlugin{Name:cmp-plugin-v1.0,Env:[]*EnvEntry{},Parameters:[]ApplicationSourcePluginParameter{},},Chart:,Ref:,} {https://kubernetes.default.svc reloader  false} repo.XXXXXXXXX &SyncPolicy{Automated:&SyncPolicyAutomated{Prune:true,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true ServerSideApply=true RespectIgnoreDifferences=true],Retry:&RetryStrategy{Limit:-1,Backoff:&Backoff{Duration:5s,Factor:*2,MaxDuration:10m,},},ManagedNamespaceMetadata:nil,} [{networking.k8s.io Ingress   [/metadata/annotations/kubernetes.io~1ingress.class] [] []} {* *   [] [] [kube-controller-manager istio-operator pilot-discovery]} {apps Deployment   [] [] [vpa-recommender]} {admissionregistration.k8s.io *   [/webhooks/0/clientConfig/caBundle] [] [cainjector]} {admissionregistration.k8s.io *   [/webhooks/0/failurePolicy] [] [pilot-discovery]} {admissionregistration.k8s.io *   [] [] [cainjector]} {* ConfigMap   [] [] [wf-istio-ingress-operator]} {apps StatefulSet   [/spec/volumeClaimTemplates/0/apiVersion /spec/volumeClaimTemplates/0/kind /spec/volumeClaimTemplates/1/apiVersion /spec/volumeClaimTemplates/1/kind] [] []} { Service   [/spec/ports/0/appProtocol /spec/ports/1/appProtocol /spec/ports/2/appProtocol /spec/ports/3/appProtocol /spec/ports/4/appProtocol] [] []} {apps Deployment istiod-1-19  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []} {apps DaemonSet istio-cni-node  [] [.spec.template.spec.containers[].env[] | select(.name == \"GOMAXPROCS\" or .name == \"GOMEMLIMIT\") | .valueFrom.resourceFieldRef.divisor] []}] [] <nil> []}" type=Normal
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="Update successful" application=argocd/reloader
07-01 13:53:35 time="2024-07-01T17:53:35Z" level=info msg="Reconciliation completed" application=argocd/reloader dedup_ms=0 dest-name= dest-namespace=reloader dest-server="https://kubernetes.default.svc" diff_ms=9 fields.level=1 git_ms=7 health_ms=0 live_ms=133 patch_ms=17 setop_ms=0 settings_ms=0 sync_ms=0 time_ms=336
pasha-codefresh commented 2 months ago

This occurs because, during the reconciliation loop, ArgoCD checks whether to execute self-healing using the condition: alreadyAttempted && (!selfHeal || !attemptPhase.Successful()).

The "already attempted" state is determined by the condition app.Status.OperationState.SyncResult.Revision != commitSHA, which is different for the reasons described in the issue.

I have PR draft, i will open it in coming day or two.

pasha-codefresh commented 1 month ago

During testing we discovered issue with sync, i am continue working on it, sorry for delay

Ezzahhh commented 1 month ago

I believe my PR may have solved this? https://github.com/argoproj/argo-cd/pull/19512

pasha-codefresh commented 1 month ago

Unfortunately no @Ezzahhh . I have found edge case that i am testing now, i really hope to open draft till EOW, i didnt forget about this issue. I just want to make sure that it works before open PR