2.6 RC2: ApplicationSet RollingSync Deploys all Apps at once

[christianh814]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Trying out the new RollingSync alpha feature isn't working as expected.

When I try to use this feature, it seems that the ApplicationSet Controller creates them all at once (instead of one at a time as my configuration suggests)

To Reproduce

1. Install 2.6 RC2
2. Apply my example: kubectl apply -f https://raw.githubusercontent.com/christianh814/gitops-examples/main/applicationsets/rollingsync/appset-rollingsync.yaml

An example of the config I used:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: pricelist
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - srv: config
        path: applicationsets/rollingsync/apps/pricelist-config
      - srv: db
        path: applicationsets/rollingsync/apps/pricelist-db
      - srv: frontend
        path: applicationsets/rollingsync/apps/pricelist-frontend
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: srv
              operator: In
              values:
                - config
        - matchExpressions:
            - key: srv
              operator: In
              values:
                - db
        - matchExpressions:
            - key: srv
              operator: In
              values:
                - frontend
  template:
    metadata:
      name: 'pricelist-{{srv}}'
      labels:
        pricelist-component: '{{srv}}'
    spec:
      project: default
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 5
          backoff:
            duration: 5s
            maxDuration: 3m0s
            factor: 2
      source:
        repoURL: https://github.com/christianh814/gitops-examples
        targetRevision: main
        path: '{{path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: pricelist

Expected behavior

I expected the ApplicationSet controller to create the Applications as needed/defined in the config. In my case, I expected the first Application to deploy and finish before the second one is even created, and so on.

Screenshots

All applications appeard all at once, instead of one at a time

Version

$ argocd version
argocd: v2.5.3+0c7de21
  BuildDate: 2022-11-28T17:11:59Z
  GitCommit: 0c7de210ae66bf631cc4f27ee1b5cdc0d04c1c96
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.6.0-rc2+6a9f37c
  BuildDate: 2023-01-05T15:10:14Z
  GitCommit: 6a9f37ca7d9cabfde298d69e8731874e3e5347cf
  GitTreeState: clean
  GoVersion: go1.18.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.3+g835b733
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

I see the following, which indicates every app being created at once.

time="2023-01-09T19:10:56Z" level=info msg="generated 3 applications" generator="{&ListGenerator{Elements:[]JSON{{[123 34 112 97 116 104 34 58 34 97 112 112 108 105 99 97 116 105 111 110 115 101 116 115 47 114 111 108 108 105 110 103 115 121 110 99 47 97 112 112 115 47 112 114 105 99 101 108 105 115 116 45 99 111 110 102 105 103 34 44 34 115 114 118 34 58 34 99 111 110 102 105 103 34 125]},{[123 34 112 97 116 104 34 58 34 97 112 112 108 105 99 97 116 105 111 110 115 101 116 115 47 114 111 108 108 105 110 103 115 121 110 99 47 97 112 112 115 47 112 114 105 99 101 108 105 115 116 45 100 98 34 44 34 115 114 118 34 58 34 100 98 34 125]},{[123 34 112 97 116 104 34 58 34 97 112 112 108 105 99 97 116 105 111 110 115 101 116 115 47 114 111 108 108 105 110 103 115 121 110 99 47 97 112 112 115 47 112 114 105 99 101 108 105 115 116 45 102 114 111 110 116 101 110 100 34 44 34 115 114 118 34 58 34 102 114 111 110 116 101 110 100 34 125]},},Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},} nil nil nil nil nil nil nil nil}"
time="2023-01-09T19:10:56Z" level=info msg="unchanged Application" app=pricelist-config appSet=pricelist
time="2023-01-09T19:10:56Z" level=info msg="unchanged Application" app=pricelist-db appSet=pricelist
time="2023-01-09T19:10:56Z" level=info msg="unchanged Application" app=pricelist-frontend appSet=pricelist
time="2023-01-09T19:10:56Z" level=info msg="end reconcile" applicationset=argocd/pricelist requeueAfter=0s

[buffstyles]

@christianh814 are you sure you have enabled this feature in the appset-controller settings? https://argo-cd.readthedocs.io/en/latest/operator-manual/applicationset/Progressive-Rollouts/#enabling-progressive-rollouts

[christianh814]

@buffstyles

Admittedly, I've forgotten this. But I did enable it to re-test

$ kubectl -n argocd exec -it argocd-applicationset-controller-5d55f55c44-8s4td -- env | grep PROG
ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_PROGRESSIVE_ROLLOUTS=true

However, the similar thing happens. All applications are created at once (why?)...although now nothing is syncing? It just stays as "missing" and "outofsync"

$ kubectl get applicationsets.argoproj.io -n argocd
NAME        AGE
pricelist   8m5s
$ kubectl get applications.argoproj.io -n argocd
NAME                 SYNC STATUS   HEALTH STATUS
pricelist-config     OutOfSync     Healthy
pricelist-db         OutOfSync     Missing
pricelist-frontend   OutOfSync     Missing

Interestingly enough, my autosync is disabled?

$ kubectl export application pricelist-config -n argocd -o yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  labels:
    pricelist-component: config
  name: pricelist-config
  namespace: argocd
spec:
  destination:
    namespace: pricelist
    server: https://kubernetes.default.svc
  project: default
  source:
    path: applicationsets/rollingsync/apps/pricelist-config
    repoURL: https://github.com/christianh814/gitops-examples
    targetRevision: main
  syncPolicy:
    retry:
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m0s
      limit: 5

So in the doc, it seems it says

RollingSync will force all generated Applications to have autosync disabled.

Is there a way to change that? The fact that this is the default is disappointing. This should be configurable if it isn't.

logs

time="2023-01-12T20:26:32Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/pricelist-config
time="2023-01-12T20:26:32Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: pricelist)" application=argocd/pricelist-config
time="2023-01-12T20:26:32Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/pricelist-db
time="2023-01-12T20:26:32Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: pricelist)" application=argocd/pricelist-db
time="2023-01-12T20:26:32Z" level=info msg="Refreshing app status (spec.source differs), level (3)" application=argocd/pricelist-frontend
time="2023-01-12T20:26:32Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: pricelist)" application=argocd/pricelist-frontend
time="2023-01-12T20:26:33Z" level=info msg="getRepoObjs stats" application=argocd/pricelist-config build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=261 unmarshal_ms=261 version_ms=0
time="2023-01-12T20:26:33Z" level=info msg="Updated sync status:  -> OutOfSync" application=pricelist-config dest-namespace=pricelist dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
time="2023-01-12T20:26:33Z" level=info msg="getRepoObjs stats" application=argocd/pricelist-db build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=259 unmarshal_ms=259 version_ms=0
time="2023-01-12T20:26:33Z" level=info msg="Updated health status:  -> Healthy" application=pricelist-config dest-namespace=pricelist dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
time="2023-01-12T20:26:33Z" level=info msg="Updated sync status:  -> OutOfSync" application=pricelist-db dest-namespace=pricelist dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
time="2023-01-12T20:26:33Z" level=info msg="Updated health status:  -> Missing" application=pricelist-db dest-namespace=pricelist dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
time="2023-01-12T20:26:33Z" level=info msg="Update successful" application=argocd/pricelist-config
time="2023-01-12T20:26:33Z" level=info msg="Reconciliation completed" application=argocd/pricelist-config dedup_ms=0 dest-name= dest-namespace=pricelist dest-server="https://kubernetes.default.svc" diff_ms=0 fields.level=3 git_ms=261 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=267
time="2023-01-12T20:26:33Z" level=info msg="getRepoObjs stats" application=argocd/pricelist-frontend build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=264 unmarshal_ms=264 version_ms=0
time="2023-01-12T20:26:33Z" level=info msg="Updated sync status:  -> OutOfSync" application=pricelist-frontend dest-namespace=pricelist dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
time="2023-01-12T20:26:33Z" level=info msg="Update successful" application=argocd/pricelist-db
time="2023-01-12T20:26:33Z" level=info msg="Reconciliation completed" application=argocd/pricelist-db dedup_ms=0 dest-name= dest-namespace=pricelist dest-server="https://kubernetes.default.svc" diff_ms=1 fields.level=3 git_ms=259 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=268
time="2023-01-12T20:26:33Z" level=info msg="Updated health status:  -> Missing" application=pricelist-frontend dest-namespace=pricelist dest-server="https://kubernetes.default.svc" reason=ResourceUpdated type=Normal
time="2023-01-12T20:26:33Z" level=info msg="Update successful" application=argocd/pricelist-frontend
time="2023-01-12T20:26:33Z" level=info msg="Reconciliation completed" application=argocd/pricelist-frontend dedup_ms=0 dest-name= dest-namespace=pricelist dest-server="https://kubernetes.default.svc" diff_ms=0 fields.level=3 git_ms=264 health_ms=0 live_ms=0 settings_ms=0 sync_ms=0 time_ms=27

[morey-tech]

• I also forgot to enable the feature when initially testing. I wonder if having a warning in the ApplicationSet controller logs when strategy is found in the spec would help address this recurring problem for users.
• Adding an explanation to the docs on why the RollingSync strategy disables autosync on Applications would help. edit: From slack conversation; the auto-sync logic is in the ApplicationSet controller that's why it's disabled on the Applications. Worth clarifying in the docs.
• Am I correct in assuming that the initial creation of Applications from an ApplicationSet using the RollingSync strategy does not follow the strategy? If so, this should be mentioned in the docs.

Happy to take on the docs updates once confirmed.

[crenshaw-dev]

Am I correct in assuming that the initial creation of Applications from an ApplicationSet using the RollingSync strategy needs to be done manually and does not follow the strategy? If so, this should be mentioned in the docs.

I believe the appset controller should create the Applications all at once, but it should sync them according to the sync strategy.

[christianh814]

also tagging @wmgroot

[christianh814]

To recap

• Expectation is that you can have Autosync with an ApplicationSet using the RollingSync strategy. (How that happens is not a concern as long as it can be done)
• Expectation is that the inital creation of an an ApplicationSet using the RollingSync strategy should create the Applications "one at a time" until they are all synced.

The first bullet point is the most pressing IMHO

[wmgroot]

To clarify, you can have syncPolicy.automated in your Application template spec, but it will be removed by the applicationset-controller when the Applications are generated since the applicationset-controller is handling the sync timing instead. I think updating the documentation to make this clear is a good idea.

In your example, I believe that the frontend Application should not begin syncing until the db Application has returned to Healthy after Progressing. I'll see if I can reproduce the issue using your example and some dummy pods.

[wmgroot]

The unexpected behavior is happening because the RollingSync strategy is selecting none of the generated Applications.

time="2023-01-13T23:31:15Z" level=info msg="ApplicationSet pricelist step list:"
time="2023-01-13T23:31:15Z" level=info msg="step 1: []"
time="2023-01-13T23:31:15Z" level=info msg="step 2: []"
time="2023-01-13T23:31:15Z" level=info msg="step 3: []"

The matchExpressions rely on selecting the generated Applications by label in order to be compatible with all types of ApplicationSet generator.

Please try updating your matchExpression from

        - matchExpressions:
            - key: srv
              operator: In
              values:
                - config

to

        - matchExpressions:
            - key: pricelist-component
              operator: In
              values:
                - config

I'm adding some additional logging to make this more clear, such as including the step the Application belongs to in the status of the ApplicationSet.

[christianh814]

Ah okay...also the documentation could be clearer on this.

Cc @morey-tech

[wmgroot]

I can see that the docs are very ambiguous. I'll put in a PR to have logging and docs updated to make the examples clear on this expectation.

[wmgroot]

Opened https://github.com/argoproj/argo-cd/pull/12103 with a bugfix and updates to the documentation based on this issue.

[crenshaw-dev]

@christianh814 and @morey-tech do either of y'all feel comfortable giving the PR a first pass, or at least testing locally? I'm pretty busy with 2.6 bugfixes.

[christianh814]

@crenshaw-dev I'll have some time tomorrow (Jan 26th) to test it

[kevinetore]

@christianh814 @wmgroot I have 2 similar questions in regard to this issue:

Environment

• K3S cluster running argocd v2.7.6+00c914a.dirty
• Platform linux/arm64

Same as the initial issue described by Christian is happening on my side:

When I try to use this feature, it seems that the ApplicationSet Controller creates them all at once (instead of one at a time as my configuration suggests).

Applicationset

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: extensions
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: git@github.com:kevinetore/my-git-repo.git
        revision: HEAD
        directories:
          - path: core/*
          - path: extensions/*
          - path: extensions/camunda
            exclude: true
          - path: extensions/mongodb
            exclude: true
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
          - key: type
            operator: In
            values:
              - core
        - matchExpressions:
          - key: type
            operator: In
            values:
              - extensions
  template:
    metadata:
      name: '{{path.basename}}'
      labels: 
        # This will either be "core" or "extensions". I've also verified this and I can see that the generated Application resource has the correct type/label.
        type: '{{path[0]}}'
    spec:
      project: default
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{path.basename}}'
      syncPolicy:
        syncOptions:
          - ServerSideApply=true
        automated:
          prune: true
          selfHeal: true
      ignoreDifferences:
        - group: ""
          kind: Service
          name: longhorn-frontend
          jsonPointers:
            - /spec/ports/0/nodePort
      sources:
        - repoURL: git@github.com:kevinetore/my-git-repo.git
          targetRevision: HEAD
          path: '{{path}}'

Progressive syncs enabled

kubectl -n argocd exec -it pod/argocd-applicationset-controller-7786cb7547-pvdcv -- env | grep SYNC                           
ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_PROGRESSIVE_SYNCS=true

I can see that with the applicationset code mentioned above, the ApplicationSet Controller creates them all at once. Also, the Auto sync option seems to get disabled.

If needed I can create a new issue, but before I do I thought perhaps I'm missing something.

UPDATE 1

According to earlier comments, the applicationset controller should create all the Application resources simultaneously. My assumption there was incorrect.

Because the applicationset controller is now in control of when Application resources are being synced, the auto policy is, in fact, being removed. The reason for that is not yet fully clear to me, perhaps something which could be made customizable, or would it make no sense as to when to use progressive syncs or auto syncs for ArgoCD? When using progressive syncs, is there then no periodic reconciliation?

[crenshaw-dev]

The reason for that is not yet fully clear to me, perhaps something which could be made customizable, or would it make no sense as to when to use progressive syncs or auto syncs for ArgoCD?

If auto-sync were enabled, the argocd-application-controller would immediately sync all apps when any change is observed in the source repo. That would defeat the ordering mechanism provided by ApplicationSet. So when you enable Progressive Syncs, the argocd-applicationset-controller disables auto-sync and instead triggers syncs when appropriate, in the defined order.

When using progressive syncs, is there then no periodic reconciliation?

Periodic reconciliation still occurs. The argocd-application-controller notices a change in the source repo and marks the Application as out of sync. The argocd-applicationset-controller notices that the Application is out of sync and then follows the defined progressive sync order to bring that app back into sync.

[wmgroot]

I wonder if it would be more intuitive for the ProgressiveSync to respect the configuration of the automated field instead. If automated is present, it performs automatic ordered syncs. Otherwise, it does not.

The only gotcha with that approach is that automated.selfHeal wouldn't be respected because the applicationset-controller has no way to know if an OutOfSync state was caused by a git-side or k8s-api-side change.

[crenshaw-dev]

So only the Updates would be progressive, syncs would be automated? I think it makes sense.

selfHeal is a bummer... might have to consider a differently-placed field.