Support customizing default alertmanager templates

[stevesg]

Is your feature request related to a problem? Please describe.

The default templates for alertmanager are inherited from upstream and therefore hardcoded into the binary, making them hard to modify without rebuilding Mimir. There are at least two use cases where being able to customize these templates could be valuable for an operator of Mimir Alertmanager:

• Apply default branding or theming to the templates
• Add or remove default links in notifications
• Change the default links to point at a Grafana instance instead of the Alertmanager UI

For example - to change the links to the Alertmanager UI, this could be put in the template:

{{ define "__alertmanagerURL" }}https://grafana.example.com/alerting/groups{{ end }}

Describe the solution you'd like

A new optional configuration option to specify a template file. This template would be loaded after the default templates, but before user templates. Effectively this just means the "operator"/"override" template will be inserted into the template files passed here, before all other user templates: https://github.com/grafana/mimir/blob/9cc3f40a2da5a7488926a8841b34b1f40a34a56b/pkg/alertmanager/alertmanager.go#L324

Describe alternatives you've considered

• Implement the feature upstream: The need for such a feature doesn't exist in upstream Alertmanager, as it is single tenanted, there is no concept of operator. The user templates would just be used to achieve the same goal.
• Support replace the default templates entirely: This would mean the templates have to be updated each time upstream Alertmanager changes with respect to what templates are expected to exist, e.g. a new receiver is added.

[pstibrany]

Seems reasonable to me, would you like to send a PR implementing this?

[rgarrigue]

I'm also interested in this one, can someone push it forward ?