Notifications controller doesn't try to send notifications again if there's been failure reaching the destination thinking the notification has been sent already

[andrii-korotkov-verkada]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Notifications controller tried to send on deployed signal to the lambda but failed to reach it, which has an error logged. However, on next tries it thinks that the notification has been sent already.

To Reproduce

Configure an invalid destination for notifications and sync the app.

Expected behavior

The controller attempts to send the notification again.

Screenshots

Version

A custom build from recent master + one rbac PR, i.e. 2.14.0-custom-rc.

Logs

See the screenshot above.

[andrii-korotkov-verkada]

Related code https://github.com/argoproj/notifications-engine/blob/2fef5c9049fdc81910be81bdcfbe89ba39ebe87f/pkg/controller/controller.go#L204-L250

[andrii-korotkov-verkada]

Ah, actually it may be not a bug, since it's different conditions. I can see in the logs additional attempts to send on-sync-failed. Let me look more.

[andrii-korotkov-verkada]

Updated screenshot to a proper one.

[andrii-korotkov-verkada]

So it's indeed a bug, it tries to notify after trigger condition was set to false, but then it doesn't change, so no new attempts are made.

[andrii-korotkov-verkada]

Btw, looks like retries are configurable, so this can be helpful https://github.com/argoproj/notifications-engine/blob/master/pkg/services/webhook.go#L87-L89

[andrii-korotkov-verkada]

If anyone picks this up, you can test ArgoCD with your fork's version of notifications engine by using replace directive in go.mod, pointing at your branch and running go mod tidy. Just submit ArgoCD PR as draft first.

[jaehanbyun]

I want to working on it !