Expected value for ruler external_url

[james-callahan]

Currently for the ruler external_url I tried putting in the URL of our Grafana installation: e.g. external_url: "https://grafana.example.com". This bubbled through to alertmanager so that the "Source" link leads to https://grafana.example.com/graph?g0.expr=%28sum+by%28thing%29%28count_over_time%28%7Bnamespace%3D%22foo%22%7D+%7C+json+%7C+bar%3D%22maxRetries%22%5B5m%5D%29%29+%3E+0%29&g0.tab=1, which isn't valid.

This url templating (via "/graph?g0.expr=%s&g0.tab=1") appears to be coming via TableLinkForExpression in vendor/github.com/prometheus/prometheus/util/strutil/strconv.go. I don't think this url format makes any sense for Loki ruler alerts.

--> What should be in the the external_url field? My first thought is that it should bring up the Grafana Explore view.

[cstyan]

@dannykopping any thoughts here? AFAICT we're not setting this value for our loki clusters.

In our Cortex clusters we set external_url to the Prometheus instance we have running in that k8s cluster.

[dannykopping]

It's a good question. It should probably go directly to Explore as @james-callahan suggests. I checked one of our alerts sent by the ruler and the source is indeed useless when going to /graph?g0.expr=%s&g0.tab=1.

[stale[bot]]

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

[james-callahan]

Anyone got ideas on how to address this?

[pschulten]

One could rewrite the expression to grafana-syntax in the fronting http server, e.g. with nginx and lua. But this is probably super complicated.

I'm also very interested. We use Cortex as a remote_write target for multiple Prometheis and the same thing applies there, as well.

I think, I'll just rewrite the /graph.* to something generic like https://grafana.example.com/alerting/list for now, which is maybe more helpful than a broken link.

[parkedwards]

yeah we've run into this as well - right now, we've had to leave external_url undefined (and since there isn't a dedicated Loki UI, as there is with Prometheus, we don't have a ton of options as to where we might direct recipients to)

additionally, the way that alertmanager auto-affixes the path route /graph makes it so that we can't send folks to a landing page....especially if /graph doesn't exist in the route tree :(

[chewrocca]

Mimir has grafanaExploreURL in 2.6.0-rc.0. Do you think that could be implemented?

[ChristianOps]

We're having this issue too and trying to find a way to work around this problem.

Not sure what may be different in loki-ruler compared to alert rules added in Grafana managed alerts that also uses our external Alertmanager. Links are properly linking to "Alert rules / View rule" page in Grafana and they are both using the same receiver template in Alertmanager (with GeneratorURL set). Loki-ruler should be able to achieve the same unless I'm missing something here? Anyone knows how that works?

[stringang]

I found the working mechanism of external_url in the source code of version v2.8.2, and it works with recording rules.

https://github.com/grafana/loki/blob/9f809eda70babaf583bdf6bf335a28038f286618/pkg/ruler/base/ruler.go#L79-L82

[lyz-code]

Did anyone get around this issue?

[Rohlik]

We also came across this issue and we are waiting for some fix/workaround.