search

prometheus / alertmanager

Prometheus Alertmanager
https://prometheus.io
Apache License 2.0
6.59k stars 2.14k forks source link

Feature request: support GET for webhook receiver #2407

Closed jacekn closed 3 years ago

jacekn commented 3 years ago

I'd like to request a feature for the webhook receiver to support GET method. There is a situation where POST is problematic and a config option to send simple GET without payload would be helpful in those situations.

What did you do?

I wanted to create "dead man's switch" alert for prometheus and alertmanager. To do that I created an alert that is always firing. Then I added a webhook receiver that pings external service over HTTP continuously (UptimeRobot in my case). The idea is that if pings stop 3rd party service will notify us about it.

What did you expect to see?

I expected the heartbeat to be updated periodically by alertmanager.

What did you see instead? Under which circumstances?

Alertmanager gets 404 from the webhook. That's because the service expects simple GET without parameters.

Environment

0.20.0

2.11.1

simonpasquier commented 3 years ago

As described in the documentation, Alertmanager sends JSON-formatted payload to the webhook endpoint. I don't see how it would be practical with a GET. I'd rather suggest UptimeRobot to develop an integration for Alertmanager.

jacekn commented 3 years ago

As described in the documentation, Alertmanager sends JSON-formatted payload to the webhook endpoint. I don't see how it would be practical with a GET. I'd rather suggest UptimeRobot to develop an integration for Alertmanager.

Even if UptimeRobot develop Alertmanager integration (or accept POST requests but discard payload) there can still be other services, both public and private, that don't support POST. I agree that ideally every service should have good Alertmanager support but there is no way to guarantee that.

Adding a config option to the webhook integration feels like a small change with narrow scope so I was hoping that this could be done in Alertmanager. This functionality would help those who want or need to integrate with services that lack good alertmanger support.

brian-brazil commented 3 years ago

The webhook is intended to be sent to something that is purposefully written to accept and deal with the notification format, not a generic HTTP request formatter. If the remote end doesn't have support, you can always write a webhook receiver that can pass on a query that the 3rd party system can deal with.

jacekn commented 3 years ago

The webhook is intended to be sent to something that is purposefully written to accept and deal with the notification format, not a generic HTTP request formatter. If the remote end doesn't have support, you can always write a webhook receiver that can pass on a query that the 3rd party system can deal with.

Yes it's possible to create small middleware but this adds complexity to the monitoring stack. With such critical system it's better to have as few moving parts as possible as every component increases chances of false positives. However if you're not willing to make the webhook receiver more generic and flexible I may need to resort to something like that.

roidelapluie commented 3 years ago

Customization is discussed in https://github.com/prometheus/alertmanager/issues/701

I am not against changing the content, but switching to GET does not really make sense here, as Simon stated.

trallnag commented 3 years ago

@jacekn In my opinion there is no big difference between a single "non-moving" monolith that tries to cover everything or several separate services like Prometheus, Alertmanager, Grafana, middlewares, reverse proxies and so on doing their own thing. You are adding complexity nonetheless.

If you don't want to maintain another container that runs 24/7 go for a serverless function. Something according to your specific requirements could be implemented in Golang / Python on AWS Lambda in a few lines of code.

Adding a config knob to Alertmanager is a small change in itself but multiply it by a few dozen and it gets out of hand. I have several ideas for such additional knobs... how often should a request be retried? fallback url in case error is returned, expansion of variables in the url and so on

jacekn commented 3 years ago

@jacekn In my opinion there is no big difference between a single "non-moving" monolith that tries to cover everything or several separate services like Prometheus, Alertmanager, Grafana, middlewares, reverse proxies and so on doing their own thing. You are adding complexity nonetheless.

@trallnag what I meant was that alertmanager->webhook is simpler than alertmanager->middleware->webhook. middleware needs to be sufficiently HA, reachable from all subnets where alertmanger runs and so on. It's the same principal I use with prometheus itself - scraping services directly is better than going through a middleware because sometimes you end up picking up problems with middleware.

If you don't want to maintain another container that runs 24/7 go for a serverless function. Something according to your specific requirements could be implemented in Golang / Python on AWS Lambda in a few lines of code.

Yes we may need to do that, thanks for suggestions.

Adding a config knob to Alertmanager is a small change in itself but multiply it by a few dozen and it gets out of hand. I have several ideas for such additional knobs... how often should a request be retried? fallback url in case error is returned, expansion of variables in the url and so on

Yes I can see your point. Some knobs might be useful but I can also see how this is not necessarily a problem alertmanager needs to solve.

Thank you all for your replies. If the team's consensus is that this feature should not be added feel free to close this request.

simonpasquier commented 3 years ago

Thanks for the healthy discussion. I'm closing it for now.

middleware needs to be sufficiently HA, reachable from all subnets where alertmanger runs and so on.

A good practice is to run your webhook handler close/next to each Alertmanager.