Argo rollouts notifications does not work with telegram

[mishanchus]

Checklist:

• [x] I've included steps to reproduce the bug.
• [x] I've included the version of argo rollouts.

Describe the bug I'm not sure if this is a bug or if I'm doing everything right. I'm trying to set up notifications with Telegram. Using the following configuration:

apiVersion: v1
kind: Secret
metadata:
  name: argo-rollouts-notification-secret
stringData:
  telegram-token: "xxxxxxxx"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-notification-configmap
data:
  service.telegram: |
    token: $telegram-token
  template.rollout-completed: |
    message: Rollout {{.rollout.metadata.name}} has been completed.
  trigger.on-rollout-completed: |
    - send: [rollout-completed]
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: xxx
  annotations:
    notifications.argoproj.io/subscribe.on-rollout-completed.telegram: "-xxx"

To Reproduce You must have a telegram bot and use its parameters as in the configuration above.

Expected behavior After a successful Rollout, the message is sent to Telegram.

Version quay.io/argoproj/argo-rollouts:v1.6.0

Logs

time="2023-09-22T11:49:33Z" level=error msg="Notifications failed to send for eventReason RolloutCompleted with error: trigger 'on-rollout-completed' is not configured" event_reason=RolloutCompleted namespace=xxx rollout=xxx
time="2023-09-22T11:49:33Z" level=error msg="notification error: Not Found"
time="2023-09-22T11:49:33Z" level=error msg="Notifications failed to send for eventReason RolloutCompleted with error: Not Found" event_reason=RolloutCompleted namespace=xxx rollout=xxx
time="2023-09-22T11:49:33Z" level=info msg="Trigger on-rollout-completed result: []" resource=xxx/xxx
time="2023-09-22T11:49:33Z" level=info msg="Trigger on-rollout-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-completed] false}]" resource=xxx/xxx

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity.

[kzap]

• There was an update to the notifications-engine recently that had big updates: https://github.com/argoproj/argo-rollouts/pull/3384#:~:text=Add%20option%20for%20Telegram%20to%20use%20a%20thread%20id%20when%20posting%20notification%C2%A0notifications%2Dengine%23265
• I have a PR including that update: https://github.com/argoproj/argo-rollouts/pull/3384

So unsure if this is related or just invalid config.

[andrii-korotkov-verkada]

Should template.rollout-completed section have a telegram key?