Observability: Metrics: Micrometer instrumentation and bindings

[algattik]

Feature #40 Corresponding EDC issue: https://github.com/eclipse-dataspaceconnector/DataSpaceConnector/issues/419

For automatically collecting and exposing metrics, use Micrometer as documented at Micrometer Metrics spike

Description

As an EDC developer I would like to have relevant metrics data so that I can debug malfunctions during the operations.

Acceptance Criteria

• [x] Agreement with EDC community on tooling for code instrumentation
• [x] Agreement with EDC community on essential metrics to monitor
• [x] Implement EDC support for essential metrics with selected tooling
• [x] Documentation on decisions and implementation

Tasks

• [x] Discussion with Jim to agree on the approach and on the Micrometer library dependency. Agree on which module will declare the Micrometer dependency.
• [x] ADR on metrics based on OpenTelemetry Metrics spike, Micrometer Metrics spike, JMX Metrics spike
• [x] Set up a sample showing Micrometer based metrics collection in Prometheus. The sample README file explains how to modify the config to use Azure Application Insights agent instead
• [x] Implement Micrometer binding with essential metrics such as JVM, uptime, processor. Use Spring Actuator and Java Application Insights agent source code as inspiration to determine the exact bindings
• [x] Implement Micrometer binding with Jetty and Jersey to automatically collect inbound call metrics
• [x] Implement Micrometer binding with OkHttp to automatically collect outbound call metrics

[algattik]

Runtime changes

At startup:

        var registry = Metrics.globalRegistry;
        new ClassLoaderMetrics().bindTo(registry);
        new JvmMemoryMetrics().bindTo(registry);
        new JvmGcMetrics().bindTo(registry);
        new ProcessorMetrics().bindTo(registry);
        new JvmThreadMetrics().bindTo(registry);

In core/base/src/main/java/org/eclipse/dataspaceconnector/core/CoreServicesExtension.java

 OkHttpClient client = new OkHttpClient.Builder()
                .eventListener(OkHttpMetricsEventListener.builder(Metrics.globalRegistry, "okhttp.requests").build())

In extensions/http/jetty/src/main/java/org/eclipse/dataspaceconnector/extension/jetty/JettyService.java

JettyConnectionMetrics.addToAllConnectors(server, Metrics.globalRegistry);

Sample changes

Based on agera-edc/edcplanning#6 sample

In samples/04.3-open-telemetry/docker-compose.yaml

 # Env var for consumer and producer
 OTEL_METRICS_EXPORTER: prometheus

  prometheus:
    image: prom/prometheus:v2.30.3
    container_name: prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
    ports:
      - 9090:9090

new file samples/04.3-open-telemetry/prometheus/prometheus.yml

global:
  scrape_interval: 10s
  scrape_timeout: 10s

scrape_configs:
  - job_name: services
    static_configs:
      - targets:
          - 'consumer:9464'
        labels:
          service: 'Consumer'
      - targets:
          - 'provider:9464'
        labels:
          service: 'Provider'