grafana / cortex-jsonnet

Deprecated: see https://github.com/grafana/mimir/tree/main/operations/mimir instead
Apache License 2.0
74 stars 53 forks source link

Fine tune CortexIngesterReachingSeriesLimit alert #404

Closed pracucci closed 2 years ago

pracucci commented 2 years ago

What this PR does: I propose to increased CortexIngesterReachingSeriesLimit warning threshold from 70% to 80% and critical threshold from 85% to 90%.

Reason is to try to reduce a bit the noise from this alert. With an organic grow of series, it takes time to do a 10% jump, so even a 10% room once the critical alert is fired should be enough to address it. On the other side, if we get a sudden jump (eg. because of a misconfiguration) we'll not have time to address it before the limit is hit anyway (so we'll hit the limit, get paged and then we'll address it).

Which issue(s) this PR fixes: N/A

Checklist