Check watch terminations from clusters loader tests

[wojtek-t]

One of the important metrics that may suggest overload of the control plane is the number of watches that are closed by kube-apiserver because they don't keep up (or watchcache itself is not keeping). We want to add a check to our tests that will be validating if this metrics is not too high.

Metrics to exercise:

• https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/cacher/metrics.go#L58
• https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/cacher/metrics.go#L94

The easiest way to do it is probably add to prometheus-based measurement, but @marseel to confirm.

[marseel]

I believe the easiest way is to add new prometheus query to GenericPrometheusQuery here: https://github.com/kubernetes/perf-tests/blob/1f00a7787f9f47bad251d4e791a0117ad6fe7e1d/clusterloader2/testing/load/modules/measurements.yaml#L122 instead of creating new measurement.

[anshulinteg]

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[marseel]

/remove-lifecycle stale

[bouaouda-achraf]

/assign

[wojtek-t]

The only missing thing now is to enable the new measurement in our tests, right?

[marseel]

Enabling measurement and then based on results possibly adding alerting to it.