ArogCD Notifications feature parity with FluxCD (v2) for Slack

[asaf400]

Summary

Currently ArgoCD notifications (from notification-controller - which is now native) are quite limited in details \ information in regards to slack notification, only specifiying that a sync started | ended | failed..

I am referring to several issues:

• The sync trigger is not specified, I would expect to have information about the trigger reason, which can be one of: Git repo update argocd image updater manual sync from Argo UI or API

• ArgoCD Does not specify what was applied during the sync, For this, I am expecting that ArgoCD would include the information about any resource that a k8s apply resulted in a state of 'created | deleted[prune] | configured', excluding 'unchanged'. It can be 'deep object information' like a change to a deployment image (related to argocd-image-updater) like .spec.template.spec.containers[].image for deployments\statefulsets, or it can be more a summarized message: 'Configmap/helloworld configured'

  This feature in a summarized version exists in FluxCD v2 slack notifications, here are some examples:

• Related to both earlier points, argocd-image-updater (v0.12.2) currently does not provider a change notification, It's only ability to update the devops | developer of the 'proposed | upcmoing' image change is via annotations within a a new k8s event on each execution of the logic loop (sync) of the argocd-image-updater. The event and the details about image tag changes in the annotations are not consumed by ArgoCD server or notification service, and users of the tool that want such capability are left to implement a custom solution, which wouldn't be perfect because as I've described, the image-updater only informs the server of new image tags for deployed images under the deployed kustomize argo app, it does not take into account the ignoreDifferences section, that I believe is the responsibility of the server\application controller, so even a custom tool that consumes the event, now also needs to take the ignoreDifferences into account..

Motivation

I recently migrated our k8s IaC repo (kustomize based + flux helm operator) from fluxcd v1 to ArgoCD, FluxCD v1 provided a slack notification with detailed information about any change that occurred, and FluxCD v2 while produces a more summarized message, still provides enough information;

here's an example (redacted info) from FluxCD v1:

it specifies which k8s resource was updated during the sync, and if there was image updates in the docker registry, what images tag were deployed.

it essentially boils down to: for each change (git | kustomzie | image | helm) Why it was triggered (event) What changed/updated

Proposal

• ArgoCD image-updater tool should update the status field of any valid argo application resource with the summary of the update, this can be implemented as a list of changes similar to the status.history field, and each element can be the exact same object as changeList from argocd-image-updater:/pkg/argocd/update.go the argo server during sync, should then use that information while 'respecting' the ignoreDifferences section to fire a trigger and send a notification.
• ArgoCD should include the status.sync.revision value by default for the slack templates sync messages
• ArgoCD Should keep track of trigger source
• ArgoCD Should 'split' the status.operationState.syncResult.resources.syncPhase.message field into a new field, which captures the status of the kubectl apply result (created | unchanged | updated), in order to allow to keep track on all changed resources, to be able to include such information in the notification trigger template.

[todaywasawesome]

Great idea.

[sushank3]

Hi, based on the feedback, I have made changes according to the enhancement proposal. I have attached the updated proposal for your review.

The first snippet uses the "fields" property (in two sections) to create a table of information, while the second snippet uses the "blocks" property to create a more structured layout.

• Which one is better, or do you prefer the other?
• I think the block format is more readable than the field format. After confirmation, I will create a pull request for this enhancement.

Enhancements done:

• Added revision.
• Added trigger source.
• Added syncResult message (created | unchanged | updated).

Snippet 1:

     "fields": [
      {
        "title": "Sync Status",
        ...
      },
      {
        "title": "Revision",
        ...
      },
      {
        "title": "OperationinitiatedBy",
        ...
      }
      ...
      ...
      ]
    }       
    ]

Snippet 2:

      "blocks": [

          {
            "type": "section",
            "text": {
              ...
            }
          }, 
          {
            "type": "section",
            "text": {
              ...
            }
          },
          {
            "type": "section",
            "text": {
              ...
            }
          }
        ]
      }
    ]

[asaf400]

Ohh my, @sushank3 That looks awesome, I'll go over those options with my team, but I see your point there..

Snippet 1 is compact, so it would result in a cleaner slack channel when there are multiple applications or updates in close proximity, so it's more dense, which is a tradeoff, since it may not be easily human readble..

Snippet 2 is more easy on the eyes, but it would result in long messages in a slack channel..

Could you share the Fork/Branch you are working on right now for these snippets? Internally we have cooked something up as well, but it only addresses my 2nd point

ArgoCD Does not specify what was applied during the sync

My teammate rewrote part of the notification template, but it's only that, so it doesn't fix the other problems: sync trigger and revision

[asaf400]

This is what my teammate was able to achieve with only modifications for the template, but because the other features required deeper golang changes we did not invest time into that, we had other more pressing matters in our argo integration.

                attachments: |
                  [{
                    "title": "{{ .app.metadata.name}}",
                    "title_link":"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
                    "color": "#18be52",
                    "fields": [
                    {
                      "title": "Sync Status",
                      "value": "{{.app.status.sync.status}}",
                      "short": true
                    },
                    {
                      "title": "Repository",
                      "value": "{{.app.spec.source.repoURL}}",
                      "short": true
                    }
                    {{range $index, $r := .app.status.operationState.syncResult.resources}}
                    ,{
                      "title": "Updated Resource",
                      "value": "{{$r.message}}",
                      "short": false
                    }
                    {{end}}
                    ]
                  }]

[sushank3]

Oh! I think I misunderstood the aspect related to the trigger source.

• Here is a link to the template file I've been experimenting with: https://github.com/sushank3/argo-cd/blob/f162fdb427f3bcad577962c5ec09dc12811f9c3a/z_my_files/argocd-notifications-cm.yaml#L46

I've made similar modifications to what your team did, but with a slight difference. Instead of displaying "Updated Resources" for each individual resource, I've opted for using a single title.

I hope this clarifies my approach.

attachments: |
        [
        {
          "mrkdwn_in": ["text", "fields"],
          "text": "*Application:* {{ .app.metadata.name}}\n",
          "color": "#18be52",
          "fields": [
          {
            "title": "Sync status:",
            "value": "{{.app.status.sync.status}}",
            "short": true
          },

          {
            "title": "Revision:",
            "value": "{{.app.status.sync.revision}}",
            "short": true
          },
          {
            "title": "Operation initiated by:",
            "value": "*username:* {{.app.status.operationState.operation.initiatedBy.username}}\n*automated:* {{.app.status.operationState.operation.initiatedBy.automated}}", 
            "short": true 
          }
          ]
        },

        {
          "color": "#18be52",
          "fields": [
            {"title": "Resources updated:"}
            {{range $index, $c := .app.status.operationState.syncResult.resources}}
            {{if not $index}},{{end}}
            {{if $index}},{{end}}
            {
              "value": "+ {{$c.message}}\n",
              "short": true
            }
            {{end}} 
            ]
        },

[asaf400]

Bump for Argo, This still needs work from Argo and Image-updater..

[asaf400]

The best my team and I have been able to achieve without modifying argocd code is this template for argo-cd (base**) chart:

spec.source.helm.valuesObject.notifications.templates.template.app-sync-succeeded:

              email:
                subject: Application {{.app.metadata.name}} has been successfully synced.
              message: |
                {{- $myList := list -}}
                {{- $images := .app.status.operationState.syncResult.source.kustomize.images -}}

                {{- range $index, $r := .app.status.operationState.syncResult.resources -}}
                    {{- if and (eq $r.status "Synced") (or (eq $r.kind "Deployment") (eq $r.kind "StatefulSet")) (hasSuffix "configured" $r.message) -}}
                        {{- $imageVersion := "unknown" -}} {{/* Default value if not available */}}

                        {{- $resourceName := $r.name -}} {{/* Resource name for matching */}}

                        {{- range $i := $images -}}
                            {{- $imageParts := splitList ":" $i -}} {{/* Split on ':' to separate name and version */}}
                            {{- $imageName := index $imageParts 0 -}} {{/* Extract the name part (before ':') */}}
                            {{- $imageNameParts := splitList "/" $imageName -}} {{/* Split name on '/' to get the last part */}}
                            {{- $finalImageName := index $imageNameParts (sub (len $imageNameParts) 1) -}} {{/* Get the actual image name */}}

                            {{- if eq $resourceName $finalImageName -}} {{/* Match resource name with image name */}}
                                {{- $imageVersion = index $imageParts 1 -}} {{/* Extract the version part (after ':') */}}
                            {{- end -}}
                        {{- end -}}

                        {{- $myList = append $myList (print $r.name " (" $imageVersion ")") -}}
                    {{- end -}}
                {{- end -}}

                {{- if eq .serviceType "slack" -}}
                    Application {{.app.metadata.name}} has been successfully synced at {{.app.status.operationState.finishedAt}}.
                    Sync operation details are available at: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true .

                    Applications Image updates:
                    {{range $name := $myList}}
                        {{- $name}},
                    {{- end -}}
                    have been updated.

                {{- else -}}
                    {{- if gt (len $myList) 0 -}}
                        Applications:
                        {{range $name := $myList}}
                            {{- $name}},
                        {{end}}
                        have been updated
                    {{- end -}}
                {{- end -}}
              slack:
                deliveryPolicy: Post
                groupingKey: ""
                notifyBroadcast: false

Using some funky go-templating and sprig wizardry🧙 we have been able to 'extract' all deployable resource that were updated in the sync cycle, and match each .app.status.operationState.syncResult.source.kustomize.images to deployment name (natively assuming k8s resource names == image repo name)

in order to print the deployed image versions, because devs already know the latest image tag they expect from CD image push, or pre-defined tagging rules (build-id, counter, etc..)

The results are as follows:

** not to be confused with argo-cd-notifications which is defunct, and is now a zombie project between life and death 🧟 Edit: Semantics