argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.92k stars 5.46k forks source link

After upgrade to 2.12.x, sync of "out-of-sync" resources no longer working. #19965

Open tromsosec opened 1 month ago

tromsosec commented 1 month ago

Checklist:

Describe the bug

Our setup uses Argo-CD applications deployed using ApplicationSets on an app of apps pattern. Applications are multisource, i.e. using git generator and Helm Chart (OCI) + Values (Git repo). We currently do not have Auto-Sync enabled.

On our development environment, our Helm Charts artifacts are created and pushed to the OCI registry using rolling tags with "-dev" suffix, e.g. 0.0.1-dev. We overwrite the artifact with same tag upon new CI build completion.

We are currently running ArgoCD 2.11.4 and after upgrading to version 2.12.x (tried 2.12.1-3), an application running a chart with version 0.0.1-dev and upon the creation of a new chart with same tag version including changes, the changes are detected by Argo (application marked Out-Of-Sync) but the resources that are marked as Out-of-Sync cannot be synced any longer using the "Sync" button. However, if we enable "Auto-Sync" for the application the resources are synced as expected.

Additionally, performing a "Sync" with resource replacement recreates the resources but of previous state, still out of sync.

When the Helm Chart is updated to a different version tag, e.g. 0.0.2-dev there is no issue. It may be related on how the artifacts are cached. If we restart the Redis pods the first sync works but from there on we have the same above described behaviour.

To Reproduce

  1. Create and deploy an ArgoCD application which uses an OCI helm chart and a values file on a git repo.
  2. Update the Helm Chart with some changes and push the chart to the OCI registry using the same tag.
  3. Confirm that ArgoCD detects new chart (force with Hard Refresh) and application is marked Out-Of-Sync.
  4. Try to perform a manual full sync of the application.

Expected behavior

Triggering a Sync using the UI should sync the resources to the latest state and not remained unapplied.

Screenshots

CP6SljStkt

Version

argocd@argo-cd-argocd-application-controller-0:~$ argocd version
argocd: v2.12.3+6b9cd82
  BuildDate: 2024-08-27T11:57:48Z
  GitCommit: 6b9cd828c6e9807398869ad5ac44efd2c28422d6
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: linux/amd64

Logs

ApplicationController logs when manual resources sync is triggered:

2024-09-17 13:07:15.092 time="2024-09-17T11:07:15Z" level=info msg="updated 'argocd-system/sampleapp' operation (phase: Running)" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:15.092 time="2024-09-17T11:07:15Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:,Prune:false,DryRun:false,SyncStrategy:&SyncStrategy{Apply:nil,Hook:&SyncStrategyHook{SyncStrategyApply:SyncStrategyApply{Force:false,},},},Resources:[]SyncOperationResource{SyncOperationResource{Group:,Kind:ConfigMap,Name:sampleapp-sampleappcatalogapi,Namespace:devsecopsref,},SyncOperationResource{Group:apps,Kind:Deployment,Name:sampleapp-sampleappcatalogapi,Namespace:devsecopsref,},},Source:nil,Manifests:[],SyncOptions:[],Sources:[]ApplicationSource{ApplicationSource{RepoURL:registry.com/appsarchitecture/devsecopsref/examples/sampleapp/helmcharts,Path:,TargetRevision:0.2.0,Helm:&ApplicationSourceHelm{ValueFiles:[$values/lab/values.yaml],Parameters:[]HelmParameter{},ReleaseName:sampleapp,Values:,FileParameters:[]HelmFileParameter{},Version:,PassCredentials:false,IgnoreMissingValueFiles:false,SkipCrds:false,ValuesObject:nil,},Kustomize:nil,Directory:nil,Plugin:nil,Chart:sampleapp-umbrella-chart/sampleapp,Ref:,},ApplicationSource{RepoURL:https://gitrepo/appsarchitecture/devsecopsref/examples/sampleapp/deployment/sampleapp-lab.git,Path:,TargetRevision:main,Helm:nil,Kustomize:nil,Directory:nil,Plugin:nil,Chart:,Ref:values,},},Revisions:[0.2.0 194a0d0df583a4f098519ecdea9e1d212e34ddf3],} {redacted-user false} [] {0 nil}}" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:15.092 time="2024-09-17T11:07:15Z" level=info msg="Comparing app state (cluster: https://cluster/redacted, namespace: devsecopsref)" application=argocd-system/sampleapp
2024-09-17 13:07:15.121 time="2024-09-17T11:07:15Z" level=info msg="GetRepoObjs stats" application=argocd-system/sampleapp build_options_ms=0 helm_ms=25 plugins_ms=0 repo_ms=0 time_ms=27 unmarshal_ms=1 version_ms=0
2024-09-17 13:07:15.192 time="2024-09-17T11:07:15Z" level=info msg=Syncing application=argocd-system/sampleapp skipHooks=true started=false syncId=00028-jmCBS
2024-09-17 13:07:15.276 time="2024-09-17T11:07:15Z" level=info msg="Tasks (dry-run)" application=argocd-system/sampleapp syncId=00028-jmCBS tasks="[Sync/0 resource /ConfigMap:devsecopsref/sampleapp-sampleappcatalogapi obj->obj (,,), Sync/0 resource apps/Deployment:devsecopsref/sampleapp-sampleappcatalogapi obj->obj (,,)]"
2024-09-17 13:07:15.277 time="2024-09-17T11:07:15Z" level=info msg="Applying resource ConfigMap/sampleapp-sampleappcatalogapi in cluster: https://cluster/redacted, namespace: devsecopsref" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
2024-09-17 13:07:15.630 time="2024-09-17T11:07:15Z" level=info msg="Applying resource Deployment/sampleapp-sampleappcatalogapi in cluster: https://cluster/redacted, namespace: devsecopsref" dry-run=client manager=argocd-controller serverSideApply=false serverSideDiff=false
2024-09-17 13:07:15.745 time="2024-09-17T11:07:15Z" level=info msg="Updating operation state. phase: Running -> Running, message: '' -> 'one or more tasks are running'" application=argocd-system/sampleapp syncId=00028-jmCBS
2024-09-17 13:07:15.745 time="2024-09-17T11:07:15Z" level=info msg="Applying resource ConfigMap/sampleapp-sampleappcatalogapi in cluster: https://cluster/redacted, namespace: devsecopsref" dry-run=none manager=argocd-controller serverSideApply=false serverSideDiff=false
2024-09-17 13:07:16.207 time="2024-09-17T11:07:16Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'configmap/sampleapp-sampleappcatalogapi unchanged'" application=argocd-system/sampleapp kind=ConfigMap name=sampleapp-sampleappcatalogapi namespace=devsecopsref phase=Sync syncId=00028-jmCBS
2024-09-17 13:07:16.207 time="2024-09-17T11:07:16Z" level=info msg="Applying resource Deployment/sampleapp-sampleappcatalogapi in cluster: https://cluster/redacted, namespace: devsecopsref" dry-run=none manager=argocd-controller serverSideApply=false serverSideDiff=false
2024-09-17 13:07:16.264 time="2024-09-17T11:07:16Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'deployment.apps/sampleapp-sampleappcatalogapi unchanged'" application=argocd-system/sampleapp kind=Deployment name=sampleapp-sampleappcatalogapi namespace=devsecopsref phase=Sync syncId=00028-jmCBS
2024-09-17 13:07:16.264 time="2024-09-17T11:07:16Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: 'one or more tasks are running' -> 'successfully synced (all tasks run)'" application=argocd-system/sampleapp syncId=00028-jmCBS
2024-09-17 13:07:16.264 time="2024-09-17T11:07:16Z" level=info msg="sync/terminate complete" application=argocd-system/sampleapp duration=1.071493944s syncId=00028-jmCBS
2024-09-17 13:07:16.295 time="2024-09-17T11:07:16Z" level=info msg="updated 'argocd-system/sampleapp' operation (phase: Succeeded)" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:16.295 time="2024-09-17T11:07:16Z" level=info msg="Partial sync operation to  succeeded" application=sampleapp dest-namespace=devsecopsref dest-server="https://cluster/redacted" reason=OperationCompleted type=Normal
2024-09-17 13:07:16.308 time="2024-09-17T11:07:16Z" level=info msg="Refreshing app status (controller refresh requested), level (3)" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:16.309 time="2024-09-17T11:07:16Z" level=info msg="Comparing app state (cluster: https://cluster/redacted, namespace: devsecopsref)" application=argocd-system/sampleapp
2024-09-17 13:07:16.330 time="2024-09-17T11:07:16Z" level=info msg="No status changes. Skipping patch" app-namespace=argocd-system app-qualified-name=argocd-system/argocd-appsetsroot application=argocd-appsetsroot project=argocd-appsroot
2024-09-17 13:07:16.765 time="2024-09-17T11:07:16Z" level=info msg="GetRepoObjs stats" application=argocd-system/sampleapp build_options_ms=0 helm_ms=230 plugins_ms=0 repo_ms=0 time_ms=456 unmarshal_ms=224 version_ms=0
2024-09-17 13:07:16.856 time="2024-09-17T11:07:16Z" level=info msg="Skipping auto-sync: most recent sync already to " app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:16.884 time="2024-09-17T11:07:16Z" level=info msg="Update successful" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp project=devsecopsref
2024-09-17 13:07:16.885 time="2024-09-17T11:07:16Z" level=info msg="Reconciliation completed" app-namespace=argocd-system app-qualified-name=argocd-system/sampleapp application=sampleapp comparison-level=3 dedup_ms=0 dest-name=lab dest-namespace=devsecopsref dest-server="https://cluster/redacted" diff_ms=68 git_ms=456 health_ms=2 live_ms=0 patch_ms=27 project=devsecopsref setop_ms=0 settings_ms=0 sync_ms=0 time_ms=576

Kubernetes ArgoCD Deployment

kubectl get pods
NAME                                                       READY   STATUS    RESTARTS   AGE
argo-cd-argocd-application-controller-0                    1/1     Running   0          130m
argo-cd-argocd-applicationset-controller-7d6bc9759-vd9ms   1/1     Running   0          130m
argo-cd-argocd-dex-server-d774895d5-zl82b                  1/1     Running   0          145m
argo-cd-argocd-notifications-controller-6dbd674d94-jfmg4   1/1     Running   0          145m
argo-cd-argocd-repo-server-78d45c48d-vpmpf                 1/1     Running   0          129m
argo-cd-argocd-repo-server-78d45c48d-w7qjt                 1/1     Running   0          129m
argo-cd-argocd-server-b4496dcbb-2fjfn                      1/1     Running   0          129m
argo-cd-argocd-server-b4496dcbb-4s6mj                      1/1     Running   0          129m
argo-cd-redis-ha-haproxy-6ffdb86584-j9466                  1/1     Running   0          118m
argo-cd-redis-ha-haproxy-6ffdb86584-jqlv8                  1/1     Running   0          118m
argo-cd-redis-ha-haproxy-6ffdb86584-jzql5                  1/1     Running   0          118m
argo-cd-redis-ha-server-0                                  3/3     Running   0          117m
argo-cd-redis-ha-server-1                                  3/3     Running   0          115m
argo-cd-redis-ha-server-2                                  3/3     Running   0          114m
andrii-korotkov-verkada commented 3 days ago

Can you share your app's manifest, please? Also, what is that warning shown in the UI? If the resources are controlled by another app as well, this might be relevant.

tromsosec commented 2 days ago

Hi, thanks for the follow up.

Below app´s manifest:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: devsecopsref
  namespace: argocd-system
spec:
  generators:
  - git:
      repoURL: https://<REMOVED>/devsecopsref/examples/sampleapp/deployment/sampleapp-lab.git
      revision: main
      files:
      - path: "**/argocd.config.yaml"
  template:
    metadata:
      name: '{{path.basename}}-devsecopsref-sampleapp'
    spec:
      project: devsecopsref
      sources:
      - chart: sampleapp-umbrella-chart/sampleapp
        helm:
          releaseName: sampleapp
          valueFiles:
          - $values/{{path.basename}}/values.yaml
        repoURL: '{{helm.chartRegistry}}/devsecopsref/examples/sampleapp/helmcharts'
        targetRevision: '{{helm.chartVersion}}'
      - ref: values
        repoURL: https://<REMOVED>/devsecopsref/examples/sampleapp/deployment/sampleapp-lab.git
        targetRevision: main      
      destination:
        name: '{{path.basename}}'
        namespace: devsecopsref
      syncPolicy:
        automated: 
          prune: false
          selfHeal: false    
  syncPolicy:
    preserveResourcesOnDeletion: true            

We use an appset with a git generator looking for argocd.config.yaml file to create individual apps. This file just includes some parameters used within the spec, namely helm.chartRegistry and helm.chartVersion. The rest is defined in the spec itself.

Regarding the Warning it is only the "OrphanedResourceWarning" which is expected as it is enabled in the project and there are some manually created resources in the same namespace.

Thanks!

andrii-korotkov-verkada commented 2 days ago

Can you also enable the debug logging and paste the logs, please? Pasting in json format can also help IMO.