Notifications should only be sent for apps that have actually changed.

[btxbtx]

Summary

I only want to be notified about app changes for apps that have changed, and nothing else.

Even if you implement only deployment notifications according to the docs, i.e.,

    trigger.on-deployed: |
      - description: Application is synced and healthy. Triggered once per commit.
        oncePer: app.status.sync.revision
        send:
        - app-deployed
        when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'

change to a single app results in the trigger firing for every app in every cluster. Supposing there is a way to modify the above implementation so that it only updates for every app in the affected cluster, my use case would not be met.

One app change, one notification.

Motivation

I have one repo which stores cluster state for many clusters, monitored by ArgoCD. When just one app for one cluster in this repo is merged to, I receive sync notifications for every single app in every single cluster. This is too noisy, and doesn't show me the information I actually want -- which app in which cluster was deployed to.

Original discussion posted here

Proposal

Perhaps a field of app. which indicates that changes were found in the last sync operaiton, i.e. app.status.change_detected == True

[daviderli614]

Hi @btxbtx , Is this problem solved? I have the same problem，expect one app change, one notification.

[btxbtx]

No @daviderli614, this has not been solved yet.

[EppO]

I ran into the same "issue" when integrating with Grafana annotations, I was expecting the annotation to get triggered only when a successful sync occurred for a specific Argo CD App that needed an actual sync but it gets triggered for any new commits in the repo of that Argo CD App, regardless if it was in a folder the App was tracking or not. So if your Git repository contains multiple Argo CD Apps (using folders structure), every single Argo CD Apps sends a notifications of "succeeded" after a Git commit.

when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'

The obvious workaround is to use one Git repository per App but it's too drastic for me. We should have an event that gets triggered when an App is doing some actual changes (i.e. some of its tracked resources were in out-of-sync state and got synced successfully).

[de-slalonde]

I'm very new to Argo, sorry if I'm way off base. To avoid on-deployed notifications from every app on every commit, what about doing this? Quick initial tests work for me, only the app(s) that have actually changed get a new app.status.operationState.finishedAt.

    trigger.on-deployed: |
      - description: Application is synced and healthy. Triggered once per commit.
        oncePer: app.status.operationState.finishedAt
        send:
        - app-deployed
        when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'

[dtotopus]

@de-slalonde , finishedAt will change on every sync even if there are no actual changes. You may actually be right. Tried to reproduce and what happens - even though argocd does refresh the contents , sync isn't triggered so finishedAt doesn't change. So notifications for unrelated apps will be sent out during manual sync or by actually modifying relevant application configuration

[Glebcher601]

@dtotopus Well, finishedAt indeed changes every time even when resource unrelated to application is changed. I still get webhook notification on every change

[dtotopus]

Do you have auto heal enabled? I know it's not the best solution, but maybe disabling it won't trigger "proper" syncing.

[Glebcher601]

I used manual sync for application so auto-heal option is disabled.

[Glebcher601]

So, I might have a solution which kinda works:

  trigger.on-sync-succeeded: |
    - description: Application syncing has succeeded
      oncePer: app.status.sync.revision
      send:
      - deploy-status
      when: app.status.operationState.phase in ['Succeeded'] && any(app.status.operationState.syncResult.resources, {.message endsWith 'configured'})

Trigger condion field is using expr. We can see if any resource was actually changed by comparing message field. Though configured might be not the only one representing change.

[m00lecule]

Faced the same issue when storing multiple Apps in one repository. Imho this feature is essential for argo-cd.

[freekdegreef]

We had this same issue with three environments for an app in the same repo triggering webhooks on changes regardless of which environment had actually changed. I tried the suggestion offered by @Glebcher601 but in our case the deployments we're always showing a 'configured' message.

What we did in the end is this which works for now:

  trigger.on-deployed-gitlab: |
    - description: Application syncing has succeeded
      oncePer: app.status.sync.revision
      send:
      - gitlab-deployment-status
      when: app.status.operationState.phase in ['Succeeded'] and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() < 2

[eimarfandino]

Is there any progress regarding this?

[maticue]

Aren't your talking about monorepo approach? https://argo-cd.readthedocs.io/en/stable/operator-manual/notifications/triggers/#avoid-sending-same-notification-too-often

When one repo is used to sync multiple applications, the oncePer: app.status.sync.revision field will trigger a notification for each commit. For mono repos, the better approach will be using oncePer: app.status.operationState.syncResult.revision statement. This way a notification will be sent only for a particular Application's revision.

[volkanakcora]

We had this same issue with three environments for an app in the same repo triggering webhooks on changes regardless of which environment had actually changed. I tried the suggestion offered by @Glebcher601 but in our case the deployments we're always showing a 'configured' message.

What we did in the end is this which works for now:

  trigger.on-deployed-gitlab: |
    - description: Application syncing has succeeded
      oncePer: app.status.sync.revision
      send:
      - gitlab-deployment-status
      when: app.status.operationState.phase in ['Succeeded'] and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() < 2

Mate, you saved me, thank you!

[smartfin]

https://argo-cd.readthedocs.io/en/stable/operator-manual/notifications/triggers/#avoid-sending-same-notification-too-often

@maticue are there any downsides of using this method for non-monorepo apps? It looks like the condition in doc is more suitable for on-deployed trigger for any application.

[imorato]

We had this same issue with three environments for an app in the same repo triggering webhooks on changes regardless of which environment had actually changed. I tried the suggestion offered by @Glebcher601 but in our case the deployments we're always showing a 'configured' message.

What we did in the end is this which works for now:

  trigger.on-deployed-gitlab: |
    - description: Application syncing has succeeded
      oncePer: app.status.sync.revision
      send:
      - gitlab-deployment-status
      when: app.status.operationState.phase in ['Succeeded'] and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() < 2

The problem we are having is as follows: We have the when rule with 'finishedAt < 5', and the pod deployment takes more than 5 minutes but the application remains in 'progressing' because it is creating some certificates. As a result, it sends the commit in progress but never the success, since the evaluation of success is not met due to the < 5.

One way to solve it is to extend it to < X... But is there any other way to mitigate the problem?

Here my config:

    trigger.on-deployed: |
      - description: Application is synced and healthy
        oncePer: app.status.sync.revision
        send: [app-deployed]
        when: app.status != nil and app.status.operationState != nil and app.status.operationState.phase in [ 'Succeeded' ] and app.status.health != nil and app.status.health.status == 'Healthy' and time.Now().Sub(time.Parse(app.status.operationState.finishedAt)).Minutes() < 30

[korostelevm]

Aren't your talking about monorepo approach? https://argo-cd.readthedocs.io/en/stable/operator-manual/notifications/triggers/#avoid-sending-same-notification-too-often

When one repo is used to sync multiple applications, the oncePer: app.status.sync.revision field will trigger a notification for each commit. For mono repos, the better approach will be using oncePer: app.status.operationState.syncResult.revision statement. This way a notification will be sent only for a particular Application's revision.

This does not change behavior

Possibly caused by this issue https://github.com/argoproj/argo-cd/issues/10679?

[kintarowins]

isn't this available already?

  trigger.sync-operation-change: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      oncePer: app.status.sync.revision

[andrii-korotkov-verkada]

Can you describe how you are using these notifications? Maybe there are some ways to improve that workflow to dedup.

[m-wynn]

We have a large monorepo of argocd applications. When any application is updated, our team has looked into sending certain notifications, such as:

• Notifying particular teams in Slack that an application they are responsible for supporting has updated recently. If they start seeing support tickets or issues related to the app, they'll likely be able to associate it with a recent deployment easily.
• Updating our Grafana dashboards for specific applications with annotations to mark deployments. This would allow teams to associate changes in metrics (e.g. a sudden increase in error rate) with a deployment.
• Marking Github commits, deployments, and pull requests with status updates that the changes have successfully gone live.

If you manage more than a handful of application manifests in one repo, this information can become unusable. If someone on my team updates app A, and I get slack notifications that 10 apps across all my clusters have been updated, the notification is as good as useless. If my grafana dashboard for app A has annotations for every time any app in the repo has been updated, it becomes much more difficult to tell when changes that actually affect app A occur.

Some of the workarounds in this thread have helped a lot, but none of them are 100%.

[andrii-korotkov-verkada]

In my company we ended up having an abstraction layer on top of ArgoCD which works with build numbers (short git sha) and keep track of statuses in the database. This takes care of deduplication. We assume that if we get a notification for the app, it's for the latest auto-deployed version.

[andrii-korotkov-verkada]

Another option that can be used is Argo Rollouts notifications on specific Rollout objects.