search

syseleven / syseleven-exporter

Export quota and usage metrics for SysEleven
Apache License 2.0
1 stars 2 forks source link

Duplicate Prometheus alerts #46

Closed baurmatt closed 3 months ago

baurmatt commented 3 months ago

Due to deployment simplicity, we run this exporter in multiple clusters within the same projects. This leads to duplicate metrics and thus wrong/duplicate alerts.

All alerts are currently created with the following schema:

sum((syseleven_xxx_used / syseleven_xxx_total) * 100) by(region, project) > 90

I suggest to change this to:

sum((max(syseleven_xxx_used)by(__name__, project, region) / max(syseleven_xxx_total)by(__name__, project, region)) * 100) by(project, region) > 90

This will give us only the largest metric of a project and region, no matter how often the metric got collected.

cstruck commented 3 months ago

lgtm