search

idealista / prom2teams

prom2teams is an HTTP server built with Python that receives alert notifications from a previously configured Prometheus Alertmanager instance and forwards it to Microsoft Teams using defined connectors
Apache License 2.0
268 stars 84 forks source link

[SUPPORT] Is it expected behavior that my teams only receives notifications every 5 min if yes where can that be adjusted? #348

Open jonathan-dev opened 2 months ago

jonathan-dev commented 2 months ago

Description

I have The Docker image idealista/prom2teams:4.2.1 Deployed and it works fine. The only thing is that messages only show up with 5min gaps between then. Although the alerts are visible in on the prometheus dashboard way earlier.

Is this behavior to be expected or where can I set the value for it an make it react quicker?

jonathan-dev commented 2 months ago

I think I figured out the reason for my delays: To prevent bots from spamming teams returns 429 Too Many Requests after receiving many request (https://learn.microsoft.com/en-us/microsoftteams/platform/bots/how-to/rate-limit#handle-http-429-responses). However this leads to unhandled exceptions Only being shown at the start up of the container and not lower down in the logs. Maybe this exception could be handled more clearly or also clarified further down the line that an timeout is used to prevent the webhook from returning 429 or whatever is happening here exactly.

And is there any way to start posting to that webhook protected by 429 responses in a more frequent way again or do we need to create a new one?