Metric: policy evaluation latency

[flavio]

The goal of this metric is to allow an operator to understand the latency introduced by the evaluation of policies.

The policy provides also some labels, that can be used to filter the metric and provide really specific data.

Metric name

kubewarden_policy_evaluation_duration_seconds

Metric type

Histogram

Use cases

• What is the latency introduced by the evaluation of Kubewarden policies?
• What is the latency introduced by policy $POLICY_NAME?
• What are the slowest policies currently active?

Labels

• policy_name: name of the policy
• resource_name: name of the resource that has been evaluated by the policy
• resouce_kind: kind of the resource that has been evaluated by the policy. Eg: Pod, Deployment, Service,...
• resource_namespace: name of the Namespace the resource being evaluated belonged to. Of course that applies only to namespaced resources
• resource_request_operation: type of the operation (CREATE, UPDATE, DELETE)
• accepted: boolean value - final outcome of the evaluation
• mutated: boolean value - set to true when the request is accepted and mutated

Deliverables

• [x] Instrument the policy-server code
• [x] Provide examples of PromQL queries that can take advantage of this metric

[jvanz]

Some possible queries using this metric:

Show the latency of the 90th percentile among all the policies in the past 10 minutes

histogram_quantile(0.9, sum by (le) (rate(kubewarden_policy_evaluation_latency_milliseconds_bucket[10m])))

Show the latency of the 90th percentile by the policy in the past 10 minutes

histogram_quantile(0.9, sum by (le, policy_name) (rate(kubewarden_policy_evaluation_latency_milliseconds_bucket[10m])))

[jvanz]

I'm adding new panels in the Grafana dashboard to use this metric. Thus, for any query that I add or change, I'll update this issue as well.