External Alertmanager source URLs can't link to Grafana explore

[iainlane]

When ruler.alertmanager.url is is set to point to an external Alertmanager, the source URLs sent are always in the format suitable for a Prometheus web UI, not a Grafana explore.

To Reproduce

Steps to reproduce the behavior:

1. Stand up an Alertmanager
2. Configure Mimir (I'm using GEM actually but this is the same on v2.8.0) with ruler.alertmanager.url pointing to that AM. Configure ruler.external.url to the base URL of your Grafana, and add Mimir as a datasource there.
3. Add some alerting rules which are triggering and sending over there
4. Open /alerting/list in the Grafana and expand the firing alert
5. Take a look at what "See graph" links to, it will be to a Prometheus format URL

Expected behavior

Mimir should be able to generate Grafana Explore URLs. Maybe it should even do this by default when sending externally?

Some related work was landed in #3849, maybe this could be used somehow. I think that made generating correct source URLs possible if you are using the internal Alertmanager, this is about feature parity for external AMs.

Additional Context

I believe the relevant call stack is this one

• https://github.com/grafana/mimir/blob/ddbb42c5d96c07d017a40d784c3aaed2a7caa158/pkg/ruler/compat.go#L296
• https://github.com/grafana/mimir/blob/ddbb42c5d96c07d017a40d784c3aaed2a7caa158/vendor/github.com/prometheus/prometheus/rules/manager.go#L1304-L1328
• https://github.com/grafana/mimir/blob/ddbb42c5d96c07d017a40d784c3aaed2a7caa158/vendor/github.com/prometheus/prometheus/util/strutil/strconv.go#L28-L31

Note the last two are vendored from Prometheus and the generator is hardcoded. I could see upstreaming the work from #3849 being one solution here (but license incompatibility, the author would need to consent) and changing Prometheus itself to support this. Alternatively, un-vendoring sendAlerts() and doing it all on the Mimir side.

[Skaronator]

I'm a bit surprised that a Grafana product doesn't support another Grafana product. There is a hacky workaround using templates and reReplaceAll within Alertmanager to replace the url, but it still is not ideal. Furthermore, the timestamp is missing.

Ref: https://github.com/grafana/grafana/discussions/42316 / https://github.com/grafana/grafana/issues/82943

[Rohlik]

I haven't tested it yet, but what about adding a link to Grafana Explore as part of notification templating within Alertmanager 🤔? It should be possible via grafanaExploreURL function, see https://grafana.com/docs/mimir/latest/references/architecture/components/alertmanager/#templating.

[Skaronator]

@Rohlik, this configuration relates to the Mimir Alertmanager, as referenced earlier by @iainlane in #3849. However, since we use Prometheus Alertmanager instead of Mimir Alertmanager, this config doesn't apply to us. As mentioned, changes need to be made in the Mimir Ruler code logic.