Send shoot alerts to user configured alertmanagers

[wyb1]

What would you like to be added:

End users of gardener should be able to configure their shoot to receive alert notifications on a custom endpoint. Once the notification is at the custom endpoint the user can send the notification to whichever channel they want (Slack, email, etc.).

Why is this needed:

This is needed so that users can receive alerts for their shoot on whatever their preferred channel is.

Proposals that won't work:

• Prometheus can send alerts to several different alertmanagers. One of these alertmanagers could be under the control of the end user. This is an elegant and simple solution except for the fact that we have 2 sets of alerts in gardener: operator & user alerts. Users should not get operator alerts and vice versa. To solve this. Prometheus must be able to send a subset of alerts to one alertmanager and another subset to the other. This is not supported in Prometheus.

Proposal:

• We will create a custom webhook receiver for the alertmanager under control of gardener. This receiver will forward the alerts to an endpoint specified by the end user. The format of these alerts should be understood by an alertmanager.

• The custom webhook should forward the data in a format that another alertmanager would understand as if it were receiving it directly from Prometheus.

[wyb1]

/area monitoring

[wyb1]

We can also consider providing slack support if the users add their Slack API url to the shoot spec under .spec.monitoring.slack. Risk could be negligible if the API URL can only post to a single channel.

[wyb1]

This is difficult to solve if operators and users have a different set of alerts. For now I will close this until something like #3 is implemented and the alerts are unified.