feature, observability: Added Prometheus Metrics & Instrumentation

[nosahama]

About this change - What it does

Add Observability to the Karapace services (registry, rest) via prometheus metrics and instrumentation.

Why this way

Using prometheus, we can take advantage of it's labels, dimensional data model, available tools and ecosystem. This provides the building blocks of adding observability into the service, with focus on extensibility and testing (unit and integration).

Caveats

• Right now, we have added a few metric types majorly for the HTTP requests, but this can be extended with more metrics for the different operations of the system, i.e. schema reader metrics, etc.
• We use the karapace prefix to act as a namespace and identifier for the service metrics.
• A sample prometheus service is added to the docker compose setup to show the pulled metrics via the prometheus.scrape_configs, the UI is shown in the screenshot below.
• To support StatsD, we can add a statsd-exporter service to the docker compose setup to test the provided mapping.

• We've added scrape jobs for both the karapace-rest and karapace-registry services, both services expose metrics via the /metrics endpoint. The screenshot below shows the local prometheus UI:

  

Follow ups

• See if we want to toggle the metrics enablement via environment variables
• Add more metrics to the service, checkout the confluent metrics for SR
• Configure buckets for the Histogram metric
• Add grafana dashboards to the docker compose setup
• Test the StatsD exporter if we truly want to still support StatsD
• Add a few prometheus recording rules and alerts

References:

• Task: EC-299
• Solves: #204, but we would leave the issue open while me migrate existing metrics and implement new ones.
• Closes: #711 #890

[github-actions[bot]]

Coverage report

This PR does not seem to contain any modification to coverable code.