Expose policy metrics via a prometheus endpoint

[flavio]

The goal is to have each PolicyServer instance expose metrics about the policies currently loaded.

The metrics should be exported as a prometheus endpoint.

TODO:

• Define the metrics to be exposed
• Understand the implications of exposing metrics via OpenTelemetry vs using a prometheus library

[ereslibre]

The PR https://github.com/kubewarden/policy-server/pull/108 already includes an answer to this question I think.

We have defined two metrics -- policy evaluation counter and policy evaluation latency.

We have settled on opentelemetry with the OTLP protocol in the policy-server side, so we can deploy the opentelemetry-collector as a sidecar along with the policy-server. Then the opentelemetry-collector can be configured to either:

1. Report this metrics somewhere else. Export them in an endpoint prometheus can scrape. Also, it can push metrics to a third party service if configured like so.
2. Push these metrics through OTLP to another opentelemetry-collector within the cluster that acts as a gateway. In this case, this opentelemetry collector is the one configured only once to push these metrics to third party services, where the opentelemetry-collector acting as a sidecar is just pushing metrics through OTLP to the collector acting as a gateway.

As we have discussed, we believe embracing opentelemetry is the way to go. Because we instrument our code once with opentelemetry-rust, and then the collectors either acting as agent, or gateway, or both, can be configured as desired. If they eventually support other backends in the future we will gain this for free.