Migrate apiserver request latency metric to `apiserver_request_sli_duration_seconds`

kubernetes-monitoring / kubernetes-mixin

A set of Grafana dashboards and Prometheus alerts for Kubernetes.

Apache License 2.0

2.11k stars 599 forks source link

Migrate apiserver request latency metric to `apiserver_request_sli_duration_seconds` #809

Open dgrisonnet opened 1 year ago

dgrisonnet commented 1 year ago

In Kubernetes 1.26 we renamed the apiserver_request_slo_duration_seconds metric to apiserver_request_sli_duration_seconds: https://github.com/kubernetes/kubernetes/pull/112679 and we are looking at removing the _slo_ metric in 1.27. As such, the mixins should be updated to use the new metric.

/cc @metalmatze

metalmatze commented 1 year ago

I wonder if a recording rule could be helpful for the migration. I remember similar things from node-exporter migrations.

dgrisonnet commented 1 year ago

Yeah, that could definitely help since it would allow using whichever metric is available.

dgrisonnet commented 1 year ago

Thinking about it again, having a recording rule to select one the metrics will introduce these potential problems:

without label aggregation, the evaluation will take a lot of time considering the sheer amount of metrics
metrics won't be as fresh as they used to be since their freshness will now depend on the evaluation period

github-actions[bot] commented 1 month ago

This issue has not had any activity in the past 30 days, so the stale label has been added to it.

The stale label will be removed if there is new activity
The issue will be closed in 7 days if there is no new activity
Add the keepalive label to exempt this issue from the stale check action

Thank you for your contributions!

metalmatze commented 1 week ago

I believe this is still valid, is it @dgrisonnet?

dgrisonnet commented 1 week ago

I believe you fixed it in https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/874 no?