search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.13k stars 2.1k forks source link

External labels not applied to alerts #7327

Open lennartack opened 6 months ago

lennartack commented 6 months ago

I have configured Prometheus with external_labels. If I understand correctly the Thanos sidecar should add these labels to metrics and rules when they are proxied to the querier. This works on my setup for metrics. However, it seems external labels are not applied to alerts. The /api/v1/alerts endpoint on the Thanos querier gives the alerts without external labels.

GiedriusS commented 6 months ago

Thanos Query also deduplicates alerts so I think that's why you don't see them. Are they visible if you removed replica labels from Query?

lennartack commented 6 months ago

@GiedriusS The only replica label I have configured is replica. The external label I'm looking for is called cluster. I don't think it is deduplication as the cluster label is applied to metrics, just not to alert instances.

lennartack commented 6 months ago

I can also see the labels in the alert definitions at /api/v1/rules?type=alert. But not the instances at /api/v1/alerts. The latter is quite important for us for viewing alert instances in Grafana per cluster.

viktordaniel commented 1 month ago

:+1: +1 on it, we have same problem.