search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.72k stars 5.4k forks source link

ApplicationSet reconcilation: requeueAfterSeconds not respected #17861

Closed ptr1120 closed 6 months ago

ptr1120 commented 6 months ago

Checklist:

Describe the bug

I use a matrix generator to react on pull requests (pullRequestGenerator) and fetch image digests from docker in a custom pluginGenerator. My PluginGenerator that fetches the docker digests logs every call from argocd. It shows that the plugin is called every minute, sometimes multiple times in a minute, and sometimes with a 3-minute gap. Due to the overuse of Api requests, I run into docker pull limits. Additionally, the frequent reconciliation causes a high CPU load and log messages.

I set requeueAfterSeconds of every generator in the matrix to 6h which does not show any effect. I already played with timeout.reconciliation, which seems to improve the frequency the generator is called, but setting this value would also affect all other apps (that are not applicationSets) in their sync behavior if I understand correctly.

To Reproduce

This is my minimized ApplicationSet

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - list:
              elements:
                - foo: 'bar'
          - plugin:
              configMapRef:
                name: image-digest-resolver-plugin-argocd
              input:
                parameters:
                  images: { }
              requeueAfterSeconds: 21600
  template:
    metadata:
      name: test
      namespace: argocd
    spec:
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        path: kustomize-guestbook
        targetRevision: 'master'
        kustomize:
          namespace: test
      destination:
        name: in-cluster
        namespace: test
      project: default
      syncPolicy:
        automated: { }
        syncOptions:
          - CreateNamespace=true
  syncPolicy:
    preserveResourcesOnDeletion: false

For reproduction, one would only need a plugin that logs every call of the controller.

Expected behavior

I can control how often/frequently a generator is being executed.

Version

I installed argocd via kustomize and have no relevant changes except application.resourceTrackingMethod: annotation and some credentials.

v2.10.7+b060053

Logs

The following is always logged before the generator is called. 

received update event from owning an application
requeue: true caused by application test

Is there any way to debug/log which event/interpretation is responsible for triggering the generator(s)?

jannfis commented 6 months ago

I think the only way to solve this issue is to cache the results of a generator for a specific time. From your logs, the requeue seems to be triggered by an update to one of the generated Applications, and the ApplicationSet controller needs to ensure that the live manifest is still in the desired state.

ptr1120 commented 6 months ago

@jannfis thanks for looking into this. The problem is that there are no changes. As seen in the AppSet I reference kustomize-guestbook from /argoproj/argocd-example-apps which has no commits in the last 3 years. And also the resulting generated application uses no dynamic stuff. But nevertheless the applicationset is executed sometimes multiple times in a minute.

rumstead commented 6 months ago

@jannfis thanks for looking into this. The problem is that there are no changes. As seen in the AppSet I reference kustomize-guestbook from /argoproj/argocd-example-apps which has no commits in the last 3 years. And also the resulting generated application uses no dynamic stuff. But nevertheless the applicationset is executed sometimes multiple times in a minute.

The update in this case is from the application resource getting updated. A way to see what has changed is to dump out the application. Eg, kubectl get application test -o yaml > before.yaml and then when you see the log again "received update event from owning an application requeue: true caused by application test" dump out the application again and perform a diff.

I added a few checks in the past to avoid useless updates to applicationsets but maybe something new is triggering them.

ptr1120 commented 6 months ago

@rumstead thanks good idea will check

rumstead commented 6 months ago

@rumstead thanks good idea will check

were you able to see anything? i am also seeing our applicationset work queue pretty high and curious if there is something new.

ptr1120 commented 6 months ago

@rumstead I compared the generated test application before and after reconciliation and can see no relevant changes. The only things that have changed are metadata.generation, metadata.resourceVersion, and status.reconciledAt. I also disabled serverside-apply and tuned some compareoptions, ignoreDifference settings, this is now my argocd-cm configMap I am using with the ApplicationSet from my initial posting.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  admin.enabled: 'true'
  application.resourceTrackingMethod: annotation
  exec.enabled: 'false'
  kustomize.buildOptions: '--enable-helm --load-restrictor LoadRestrictionsNone'
  resource.compareoptions: >-
    ignoreAggregatedRoles: true
    ignoreResourceStatusField: all
  resource.customizations.ignoreResourceUpdates.all: |-
    managedFieldsManagers:
      - kube-controller-manager
    jsonPointers:
      - /spec/replicas
  resource.customizations.ignoreResourceUpdates.argoproj.io_Application: |-
    jsonPointers:
      - /status
  resource.ignoreResourceUpdatesEnabled: 'true'
  statusbadge.enabled: 'true'
  users.anonymous.enabled: 'false'

What is the expected reconciliation frequency from my given ApplicationSet? Looking into the code I would say the matrix checks all parents and takes the lowest requeueAfterSeconds, which should be 6h in my setup.

Here a full log of one reconciliation cycle:

2024-04-20 14:00:56.485 end reconcile
2024-04-20 14:00:56.480 unchanged Application
2024-04-20 14:00:56.470 12:00:56.470 +00:00|ImageDigestResolverPlugin.Controller.GetParamsController|[INF]|Successfully generated parameters [[]]
2024-04-20 14:00:56.469 
12:00:56.469 +00:00|ImageDigestResolverPlugin.Controller.GetParamsController|[INF]|Input parameters for test []
2024-04-20 14:00:56.469 apps from generator: [{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:test GenerateName: Namespace:argocd SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[resources-finalizer.argocd.argoproj.io] ManagedFields:[]} Spec:{Source:&ApplicationSource{RepoURL:https://github.com/argoproj/argocd-example-apps.git,Path:kustomize-guestbook,TargetRevision:master,Helm:nil,Kustomize:&ApplicationSourceKustomize{NamePrefix:,NameSuffix:,Images:[],CommonLabels:map[string]string{},Version:,CommonAnnotations:map[string]string{},ForceCommonLabels:false,ForceCommonAnnotations:false,Namespace:test,CommonAnnotationsEnvsubst:false,Replicas:[]KustomizeReplica{},Patches:[]KustomizePatch{},Components:[],},Directory:nil,Plugin:nil,Chart:,Ref:,} Destination:{Server: Namespace:test Name:in-cluster isServerInferred:false} Project:default SyncPolicy:&SyncPolicy{Automated:&SyncPolicyAutomated{Prune:false,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true],Retry:nil,ManagedNamespaceMetadata:nil,} IgnoreDifferences:[] Info:[] RevisionHistoryLimit:<nil> Sources:[]} Status:{Resources:[] Sync:{Status: ComparedTo:{Source:{RepoURL: Path: TargetRevision: Helm:nil Kustomize:nil Directory:nil Plugin:nil Chart: Ref:} Destination:{Server: Namespace: Name: isServerInferred:false} Sources:[] IgnoreDifferences:[]} Revision: Revisions:[]} Health:{Status: Message:} History:[] Conditions:[] ReconciledAt:<nil> OperationState:nil ObservedAt:<nil> SourceType: Summary:{ExternalURLs:[] Images:[]} ResourceHealthSource: SourceTypes:[] ControllerNamespace:} Operation:nil}]
2024-04-20 14:00:56.469 generated 1 applications
2024-04-20 14:00:56.466 Reconciliation completed
2024-04-20 14:00:56.466 Update successful
2024-04-20 14:00:56.465 requeue: true caused by application test
2024-04-20 14:00:56.465 received update event from owning an application

By the way the log line beginning with apps from generator and printing the application object looks wrong to me, because she part Generation:0 CreationTimestamp:0001-01-01 00:00:00 is not showing correct data. Any idea how to control reconciliation frequency?

rumstead commented 6 months ago

It looks like your plugin is changing something that is causing the sync to happen right away. 

    // the applicationset controller owns the application spec, labels, annotations, and finalizers on the applications
    if !reflect.DeepEqual(appOld.Spec, appNew.Spec) ||
        !reflect.DeepEqual(appOld.ObjectMeta.GetAnnotations(), appNew.ObjectMeta.GetAnnotations()) ||
        !reflect.DeepEqual(appOld.ObjectMeta.GetLabels(), appNew.ObjectMeta.GetLabels()) ||
        !reflect.DeepEqual(appOld.ObjectMeta.GetFinalizers(), appNew.ObjectMeta.GetFinalizers()) {
        return true
    }

My guess is it is something in the spec. Do you have progressive syncs enabled?

ptr1120 commented 6 months ago

How can my plugin change something? I use exactly this ApplicationSet:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - list:
              elements:
                - foo: 'bar'
          - plugin:
              configMapRef:
                name: image-digest-resolver-plugin-argocd
              input:
                parameters:
                  images: { }
              requeueAfterSeconds: 21600
  template:
    metadata:
      name: test
      namespace: argocd
    spec:
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        path: kustomize-guestbook
        targetRevision: 'master'
        kustomize:
          namespace: test
      destination:
        name: in-cluster
        namespace: test
      project: default
      syncPolicy:
        automated: { }
        syncOptions:
          - CreateNamespace=true
  syncPolicy:
    preserveResourcesOnDeletion: false

Here I only let the plugin get called with an empty list of images as input parameters. As you can see there are no changes in the spec except the kustomize namespace.

rumstead commented 6 months ago

Are you comparing the application called test or your applicationset called test? You should be looking at the application. 

2024-04-20 14:00:56.465 requeue: true caused by application test
ptr1120 commented 6 months ago

I compared the Application, this is how it looks like before and after the update:

Before

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: "2024-04-17T14:25:18Z"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  generation: 137
  name: test
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: test
    uid: 96993792-5db0-4463-b657-3fbd968f6fe5
  resourceVersion: "1971302"
  uid: 50062d06-7de0-4eaf-be81-b4a3ac790aa7
spec:
  destination:
    name: in-cluster
    namespace: test
  project: default
  source:
    kustomize:
      namespace: test
    path: kustomize-guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: master
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true
status:
  controllerNamespace: argocd
  health:
    status: Healthy
  history:
  - deployStartedAt: "2024-04-17T14:25:18Z"
    deployedAt: "2024-04-17T14:25:18Z"
    id: 0
    revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
    source:
      kustomize:
        namespace: test
      path: kustomize-guestbook
      repoURL: https://github.com/argoproj/argocd-example-apps.git
      targetRevision: master
  operationState:
    finishedAt: "2024-04-17T14:25:19Z"
    message: successfully synced (all tasks run)
    operation:
      initiatedBy:
        automated: true
      retry:
        limit: 5
      sync:
        revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
        syncOptions:
        - CreateNamespace=true
    phase: Succeeded
    startedAt: "2024-04-17T14:25:18Z"
    syncResult:
      resources:
      - group: ""
        hookPhase: Running
        kind: Service
        message: service/kustomize-guestbook-ui created
        name: kustomize-guestbook-ui
        namespace: test
        status: Synced
        syncPhase: Sync
        version: v1
      - group: apps
        hookPhase: Running
        kind: Deployment
        message: deployment.apps/kustomize-guestbook-ui created
        name: kustomize-guestbook-ui
        namespace: test
        status: Synced
        syncPhase: Sync
        version: v1
      revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
      source:
        kustomize:
          namespace: test
        path: kustomize-guestbook
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: master
  reconciledAt: "2024-04-20T09:37:40Z"
  resources:
  - health:
      status: Healthy
    kind: Service
    name: kustomize-guestbook-ui
    namespace: test
    status: Synced
    version: v1
  - group: apps
    health:
      status: Healthy
    kind: Deployment
    name: kustomize-guestbook-ui
    namespace: test
    status: Synced
    version: v1
  sourceType: Kustomize
  summary:
    images:
    - gcr.io/heptio-images/ks-guestbook-demo:0.1
  sync:
    comparedTo:
      destination:
        name: in-cluster
        namespace: test
      source:
        kustomize:
          namespace: test
        path: kustomize-guestbook
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: master
    revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
    status: Synced

After

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: "2024-04-17T14:25:18Z"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  generation: 138
  name: test
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: test
    uid: 96993792-5db0-4463-b657-3fbd968f6fe5
  resourceVersion: "1972688"
  uid: 50062d06-7de0-4eaf-be81-b4a3ac790aa7
spec:
  destination:
    name: in-cluster
    namespace: test
  project: default
  source:
    kustomize:
      namespace: test
    path: kustomize-guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: master
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true
status:
  controllerNamespace: argocd
  health:
    status: Healthy
  history:
  - deployStartedAt: "2024-04-17T14:25:18Z"
    deployedAt: "2024-04-17T14:25:18Z"
    id: 0
    revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
    source:
      kustomize:
        namespace: test
      path: kustomize-guestbook
      repoURL: https://github.com/argoproj/argocd-example-apps.git
      targetRevision: master
  operationState:
    finishedAt: "2024-04-17T14:25:19Z"
    message: successfully synced (all tasks run)
    operation:
      initiatedBy:
        automated: true
      retry:
        limit: 5
      sync:
        revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
        syncOptions:
        - CreateNamespace=true
    phase: Succeeded
    startedAt: "2024-04-17T14:25:18Z"
    syncResult:
      resources:
      - group: ""
        hookPhase: Running
        kind: Service
        message: service/kustomize-guestbook-ui created
        name: kustomize-guestbook-ui
        namespace: test
        status: Synced
        syncPhase: Sync
        version: v1
      - group: apps
        hookPhase: Running
        kind: Deployment
        message: deployment.apps/kustomize-guestbook-ui created
        name: kustomize-guestbook-ui
        namespace: test
        status: Synced
        syncPhase: Sync
        version: v1
      revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
      source:
        kustomize:
          namespace: test
        path: kustomize-guestbook
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: master
  reconciledAt: "2024-04-20T09:40:40Z"
  resources:
  - health:
      status: Healthy
    kind: Service
    name: kustomize-guestbook-ui
    namespace: test
    status: Synced
    version: v1
  - group: apps
    health:
      status: Healthy
    kind: Deployment
    name: kustomize-guestbook-ui
    namespace: test
    status: Synced
    version: v1
  sourceType: Kustomize
  summary:
    images:
    - gcr.io/heptio-images/ks-guestbook-demo:0.1
  sync:
    comparedTo:
      destination:
        name: in-cluster
        namespace: test
      source:
        kustomize:
          namespace: test
        path: kustomize-guestbook
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: master
    revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
    status: Synced
rumstead commented 6 months ago

I copied and pasted your two manifests as a test case and it doesn't trigger an update event. Just to confirm what you did:

  1. Took a snapshot of the before application, kubectl get application test -o yaml

  2. Waited for the logs in the application controller that show

    • received update event from owning an application requeue: true caused by application test

  3. After seeing the above logs, took another snapshot of the test application kubectl get application test -o yaml

ptr1120 commented 6 months ago

Thank you @rumstead , so I can confirm doing it exactly as you described:

Before

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: "2024-04-22T15:55:35Z"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  generation: 22
  name: test
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: test
    uid: 65aea640-3cba-495b-8255-571bf44de81d
  resourceVersion: "12940"
  uid: 6537e8fe-e400-4e95-808a-d541c7832ab7
spec:
  destination:
    name: in-cluster
    namespace: test
  project: default
  source:
    kustomize:
      namespace: test
    path: kustomize-guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: master
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true
status:
  controllerNamespace: argocd
  health:
    status: Healthy
  history:
  - deployStartedAt: "2024-04-22T15:55:38Z"
    deployedAt: "2024-04-22T15:55:55Z"
    id: 0
    revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
    source:
      kustomize:
        namespace: test
      path: kustomize-guestbook
      repoURL: https://github.com/argoproj/argocd-example-apps.git
      targetRevision: master
  operationState:
    finishedAt: "2024-04-22T15:55:55Z"
    message: successfully synced (all tasks run)
    operation:
      initiatedBy:
        automated: true
      retry:
        limit: 5
      sync:
        revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
        syncOptions:
        - CreateNamespace=true
    phase: Succeeded
    startedAt: "2024-04-22T15:55:38Z"
    syncResult:
      resources:
      - group: ""
        hookPhase: Running
        kind: Namespace
        message: namespace/test created
        name: test
        namespace: ""
        status: Synced
        syncPhase: PreSync
        version: v1
      - group: ""
        hookPhase: Running
        kind: Service
        message: service/kustomize-guestbook-ui created
        name: kustomize-guestbook-ui
        namespace: test
        status: Synced
        syncPhase: Sync
        version: v1
      - group: apps
        hookPhase: Running
        kind: Deployment
        message: deployment.apps/kustomize-guestbook-ui created
        name: kustomize-guestbook-ui
        namespace: test
        status: Synced
        syncPhase: Sync
        version: v1
      revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
      source:
        kustomize:
          namespace: test
        path: kustomize-guestbook
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: master
  reconciledAt: "2024-04-22T16:34:52Z"
  resources:
  - health:
      status: Healthy
    kind: Service
    name: kustomize-guestbook-ui
    namespace: test
    status: Synced
    version: v1
  - group: apps
    health:
      status: Healthy
    kind: Deployment
    name: kustomize-guestbook-ui
    namespace: test
    status: Synced
    version: v1
  sourceType: Kustomize
  summary:
    images:
    - gcr.io/heptio-images/ks-guestbook-demo:0.1
  sync:
    comparedTo:
      destination:
        name: in-cluster
        namespace: test
      source:
        kustomize:
          namespace: test
        path: kustomize-guestbook
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: master
    revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
    status: Synced

After

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  creationTimestamp: "2024-04-22T15:55:35Z"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  generation: 23
  name: test
  namespace: argocd
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ApplicationSet
    name: test
    uid: 65aea640-3cba-495b-8255-571bf44de81d
  resourceVersion: "13530"
  uid: 6537e8fe-e400-4e95-808a-d541c7832ab7
spec:
  destination:
    name: in-cluster
    namespace: test
  project: default
  source:
    kustomize:
      namespace: test
    path: kustomize-guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: master
  syncPolicy:
    automated: {}
    syncOptions:
    - CreateNamespace=true
status:
  controllerNamespace: argocd
  health:
    status: Healthy
  history:
  - deployStartedAt: "2024-04-22T15:55:38Z"
    deployedAt: "2024-04-22T15:55:55Z"
    id: 0
    revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
    source:
      kustomize:
        namespace: test
      path: kustomize-guestbook
      repoURL: https://github.com/argoproj/argocd-example-apps.git
      targetRevision: master
  operationState:
    finishedAt: "2024-04-22T15:55:55Z"
    message: successfully synced (all tasks run)
    operation:
      initiatedBy:
        automated: true
      retry:
        limit: 5
      sync:
        revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
        syncOptions:
        - CreateNamespace=true
    phase: Succeeded
    startedAt: "2024-04-22T15:55:38Z"
    syncResult:
      resources:
      - group: ""
        hookPhase: Running
        kind: Namespace
        message: namespace/test created
        name: test
        namespace: ""
        status: Synced
        syncPhase: PreSync
        version: v1
      - group: ""
        hookPhase: Running
        kind: Service
        message: service/kustomize-guestbook-ui created
        name: kustomize-guestbook-ui
        namespace: test
        status: Synced
        syncPhase: Sync
        version: v1
      - group: apps
        hookPhase: Running
        kind: Deployment
        message: deployment.apps/kustomize-guestbook-ui created
        name: kustomize-guestbook-ui
        namespace: test
        status: Synced
        syncPhase: Sync
        version: v1
      revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
      source:
        kustomize:
          namespace: test
        path: kustomize-guestbook
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: master
  reconciledAt: "2024-04-22T16:37:52Z"
  resources:
  - health:
      status: Healthy
    kind: Service
    name: kustomize-guestbook-ui
    namespace: test
    status: Synced
    version: v1
  - group: apps
    health:
      status: Healthy
    kind: Deployment
    name: kustomize-guestbook-ui
    namespace: test
    status: Synced
    version: v1
  sourceType: Kustomize
  summary:
    images:
    - gcr.io/heptio-images/ks-guestbook-demo:0.1
  sync:
    comparedTo:
      destination:
        name: in-cluster
        namespace: test
      source:
        kustomize:
          namespace: test
        path: kustomize-guestbook
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        targetRevision: master
    revision: d7927a27b4533926b7d86b5f249cd9ebe7625e90
    status: Synced

Just the same non relevant changes.

ptr1120 commented 6 months ago

@rumstead I was able to simplify the problem with the permanent reconciliations. I removed my plugin and utilized a merge generator with two list generators.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test
  namespace: argocd
spec:
  generators:
    - merge:
        generators:
          - list:
              elements:
                - foo: 'bar'
          - list:
              elements:
                - foo: 'bar'
        mergeKeys:
          - foo
  template:
    metadata:
      name: test
      namespace: argocd
    spec:
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        path: kustomize-guestbook
        targetRevision: 'master'
        kustomize:
          namespace: test
      destination:
        name: in-cluster
        namespace: test
      project: default
      syncPolicy:
        automated: { }
        syncOptions:
          - CreateNamespace=true
  syncPolicy:
    preserveResourcesOnDeletion: false

This results in the following log

2024-04-23 10:41:40.599 end reconcile
2024-04-23 10:41:40.596 unchanged Application
2024-04-23 10:41:40.592 apps from generator: [{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:test GenerateName: Namespace:argocd SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[resources-finalizer.argocd.argoproj.io] ManagedFields:[]} Spec:{Source:&ApplicationSource{RepoURL:https://github.com/argoproj/argocd-example-apps.git,Path:kustomize-guestbook,TargetRevision:master,Helm:nil,Kustomize:&ApplicationSourceKustomize{NamePrefix:,NameSuffix:,Images:[],CommonLabels:map[string]string{},Version:,CommonAnnotations:map[string]string{},ForceCommonLabels:false,ForceCommonAnnotations:false,Namespace:test,CommonAnnotationsEnvsubst:false,Replicas:[]KustomizeReplica{},Patches:[]KustomizePatch{},Components:[],},Directory:nil,Plugin:nil,Chart:,Ref:,} Destination:{Server: Namespace:test Name:in-cluster isServerInferred:false} Project:default SyncPolicy:&SyncPolicy{Automated:&SyncPolicyAutomated{Prune:false,SelfHeal:false,AllowEmpty:false,},SyncOptions:[CreateNamespace=true],Retry:nil,ManagedNamespaceMetadata:nil,} IgnoreDifferences:[] Info:[] RevisionHistoryLimit:<nil> Sources:[]} Status:{Resources:[] Sync:{Status: ComparedTo:{Source:{RepoURL: Path: TargetRevision: Helm:nil Kustomize:nil Directory:nil Plugin:nil Chart: Ref:} Destination:{Server: Namespace: Name: isServerInferred:false} Sources:[] IgnoreDifferences:[]} Revision: Revisions:[]} Health:{Status: Message:} History:[] Conditions:[] ReconciledAt:<nil> OperationState:nil ObservedAt:<nil> SourceType: Summary:{ExternalURLs:[] Images:[]} ResourceHealthSource: SourceTypes:[] ControllerNamespace:} Operation:nil}]
2024-04-23 10:41:40.592 generated 1 applications
2024-04-23 10:41:40.592 requeue: true caused by application test
2024-04-23 10:41:40.592 received update event from owning an application
2024-04-23 10:41:40.592 Reconciliation completed
2024-04-23 10:41:40.592 Update successful
2024-04-23 10:41:40.580 Skipping auto-sync: application status is Synced
2024-04-23 10:41:40.572 GetRepoObjs stats
2024-04-23 10:41:40.572 finished unary call with code OK
2024-04-23 10:41:40.572 manifest cache hit: &ApplicationSource{RepoURL:https://github.com/argoproj/argocd-example-apps.git,Path:kustomize-guestbook,TargetRevision:master,Helm:nil,Kustomize:&ApplicationSourceKustomize{NamePrefix:,NameSuffix:,Images:[],CommonLabels:map[string]string{},Version:,CommonAnnotations:map[string]string{},ForceCommonLabels:false,ForceCommonAnnotations:false,Namespace:test,CommonAnnotationsEnvsubst:false,Replicas:[]KustomizeReplica{},Patches:[]KustomizePatch{},Components:[],},Directory:nil,Plugin:nil,Chart:,Ref:,}/d7927a27b4533926b7d86b5f249cd9ebe7625e90

The "end reconcile" log message shows the attribute requeueAfter: 0 which is correct and should never reque the application reconciliation in my understanding.

rumstead commented 6 months ago

When I debug this locally, I see two events that retrigger the applicationset to be reconciled. I wouldn't be surprised if this is multiplied by the number of applications that are created.

  1. Create
  2. Update

After both events are processed, no additional requeues happen.

ptr1120 commented 6 months ago

With the current ApplicationSet (which is the only argocd application in my test setup), it reconciles less but still every 2-3 minutes. Every time with a "requeue: true caused by application test" log message. But by definition the merge generator, should requeue never (with 2 list generators). Do we have different configurations (I am using kustomize with github.com/argoproj/argo-cd/manifests/ha/cluster-install?ref=v2.10.7 and the argocd-cm ConfigMap from above)?

ptr1120 commented 6 months ago

The issue seems not no be limited to matrix/merge generator, when using just a list generator it also reconciles every time due to requeue: true caused by application test

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - foo: 'bar'
  template:
    metadata:
      name: test
      namespace: argocd
    spec:
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        path: kustomize-guestbook
        targetRevision: 'master'
        kustomize:
          namespace: test
      destination:
        name: in-cluster
        namespace: test
      project: default
      syncPolicy:
        automated: { }
        syncOptions:
          - CreateNamespace=true
  syncPolicy:
    preserveResourcesOnDeletion: false
rumstead commented 6 months ago

There are multiple ways to trigger a reconciliation. The one you are describing is telling the controller to check a resource after a given period, eg using requeueAfter. Another way is a resource that the ApplicationSet controller manages changing, eg an ApplicationSets or an Application owned by an ApplicationSet.

You are experiencing the latter. I have let my debugger sit for 10 minutes and I haven't had any additional triggers since the initial create + update.

ptr1120 commented 6 months ago

Thats strange. To me it looks like the method shouldRequeueApplicationSet from applicationSetController returns true (requeue) due to an unexpected inequality of appOld and appNew. Do you agree? I will now check weather using the latest rc of argocd has the same behaviour.

rumstead commented 6 months ago

Oh, it is getting triggered I just wasn't patient enough. After I dug a little deeper it looks like reflect.DeepEqual doesn't treat nil == to an empty slice/map.

Thats strange. To me it looks like the method shouldRequeueApplicationSet from applicationSetController returns true (requeue) due to an unexpected inequality of appOld and appNew. Do you agree?

Agreed.

ptr1120 commented 6 months ago

Thank you very much @rumstead

rumstead commented 6 months ago

@ptr1120 want to test the changes on quay.io/argoproj/argocd:latest or ghcr.io/argoproj/argo-cd/argocd:2.11.0-1eec8eb7

It looks to be working the applicationsets / applications I am testing but want to make sure it is working for you as well. An interesting metric to monitor is the applicationset workqueue depth.

ptr1120 commented 6 months ago

Thank you @rumstead . I finished testing right now using the tag you mentioned and found out that it is working perfectly!