ArgoCD triggering of grafana annotation not working

[DeChinees]

Problem

We try to create grafana annotations based on argocd statuses' by following this guide: https://argocd-notifications.readthedocs.io/en/latest/services/grafana/ Argocd does not seem to create annotations.

We are running version Argocd 2.5.2 (latest) an several Azure Kubernetes cluster.

Connectivity was tested successfully on the argcd-notification pod with the following curl command

curl --request POST \
  --url https://grafana.internal.net/api/annotations \
  --header 'Authorization: Bearer WZ2==' \
  --header 'Content-Type: application/json' \
  --data '{
  "text": "curl created this annotation",
  "tags": ["dev-0-deployment"],
  "data": {}
}'

Configmap

kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  server.log.level: "debug"
  reposerver.log.level: "debug"
  service.grafana: |
    apiUrl: https://grafana.internal.net/api
    apiKey: <secret>

Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: dev-0-base
  labels:
    team: enabling
    environment: dev-0
  annotations:
    argocd.argoproj.io/manifest-generate-paths: .
    argocd-image-updater.argoproj.io/update-strategy: digest
    notifications.argoproj.io/subscribe.on-deployed.grafana: dev-0-base-deployment
    notifications.argoproj.io/subscribe.on-health-degraded.grafana: dev-0-base-degraded
    notifications.argoproj.io/subscribe.on-sync-failed.grafana: dev-0-base-sync-failed
    notifications.argoproj.io/subscribe.on-sync-status-unknown.grafana: dev-0-base-status-unknown
    notifications.argoproj.io/subscribe.on-sync-succeeded.grafana: dev-0-base-sync-succeeded

Actual outcome

No annotations in grafana Argocd-notification log looks like

time="2022-11-22T11:00:44Z" level=debug msg="SyncStatus up-to-date (FinishedAt=2022-11-22 10:35:40 +0000 UTC, ReconciledAt=2022-11-22 11:00:43 +0000 UTC, Observed=0001-01-01 00:00:00 +0000 UTC" app=dev-0-base
time="2022-11-22T11:00:44Z" level=debug msg="Failed to execute condition of trigger on-sync-succeeded: trigger 'on-sync-succeeded' is not configured" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=info msg="Trigger on-sync-succeeded result: []" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=debug msg="Failed to execute condition of trigger on-health-degraded: trigger 'on-health-degraded' is not configured" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=info msg="Trigger on-health-degraded result: []" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=debug msg="Failed to execute condition of trigger on-sync-failed: trigger 'on-sync-failed' is not configured" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=info msg="Trigger on-sync-failed result: []" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=debug msg="Failed to execute condition of trigger on-sync-status-unknown: trigger 'on-sync-status-unknown' is not configured" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=info msg="Trigger on-sync-status-unknown result: []" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=debug msg="Failed to execute condition of trigger on-deployed: trigger 'on-deployed' is not configured" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=info msg="Trigger on-deployed result: []" resource=argocd/dev-0-base
time="2022-11-22T11:00:44Z" level=info msg="Processing completed" resource=argocd/dev-0-base

Expected outcome

Annotation created in grafana

[joyrex2001]

This looks related to: https://github.com/argoproj/notifications-engine/issues/77

[todaywasawesome]

@DeChinees is this issue still relevant?

[ErezWeiss]

You didn't create the trigger templates.

[zentavr]

@ErezWeiss Could you point to the doc which describes how to do that? Cannot find anything related here: https://argocd-notifications.readthedocs.io/en/stable/services/slack/#templates

My Secrets:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
  namespace: sys-argocd
type: Opaque
data:
  slack-token: xxx

CM:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: sys-argocd
data:
  defaultTriggers: >-
    ["on-created","on-deleted","on-deployed","on-health-degraded","on-sync-failed","on-sync-running","on-sync-status-unknown","on-sync-succeeded"]
  service.slack: >-
    {"token":"$slack-token","username":"ArgoCD Staging
    Notifications","icon":":crossed_fingers:"}

[zentavr]

@ErezWeiss are not they (templates) bundled by default?

[zentavr]

@alexmt I guess that the doc should be more verbose about what needs to be done

[zentavr]

Got this working. My Configmap and Secrets (examples) are here: https://gist.github.com/zentavr/c0a667d5fe6f87589df9e0ae812d1e3d

P.S.: there should be a separate process called argocd-notifications.