SDK metrics: Prometheus exporter

[iRevive]

Category	Link
OpenTelemetry spec	https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/prometheus
OpenTelemetry Java implementation	https://github.com/open-telemetry/opentelemetry-java/tree/main/exporters/prometheus/src/main/java/io/opentelemetry/exporter/prometheus
Prometheus formats	https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md
Prometheus text writer	https://github.com/prometheus/client_java/blob/main/prometheus-metrics-exposition-formats/src/main/java/io/prometheus/metrics/expositionformats/PrometheusTextFormatWriter.java
Prometheus scrape handler	https://github.com/prometheus/client_java/blob/main/prometheus-metrics-exporter-common/src/main/java/io/prometheus/metrics/exporter/common/PrometheusScrapeHandler.java

Prometheus exporter works as an HTTP server that exposes metrics in Prometheus-compatible format.

A prototype: https://github.com/iRevive/otel4s/pull/5.

A few things to consider: 1) Auto-configuration mode - SDK must launch an HTTP server 2) A user should be able to use exporter routes with its own HTTP server (see example) 3) Should we make a separate module? For example, otel4s-sdk-exporter-prometheus? 4) According to the spec, we need to support only the text writer 5) Since we need to support all platforms (JVM, Scala.js, Scala Native) we will need to partially reimplement Prometheus Writer and Scrape Handler

Examples

Auto-configuration

Env configuration:

export OTEL_METRICS_EXPORTER=prometheus
export OTEL_EXPORTER_PROMETHEUS_HOST=localhost
export OTEL_EXPORTER_PROMETHEUS_PORT=9464

Code:

OpenTelemetrySdk.autoConfigured[IO](
  _.addMetricExporterConfigurer(PrometheusMetricExporterAutoConfigure[IO])
).use { otel =>
  // emulate metrics
  for {
    meter <- otel.sdk.meterProvider.get("meter")
    counter <- meter.counter[Long]("counter").create
    _ <- counter.add(1L).delayBy(1.second).foreverM
  } yield ()
}

SDK will launch a dedicated HTTP server at localhost:9464.

Manual configuration

PrometheusMetricExporter.builder[IO].build.flatMap { exporter =>
  OpenTelemetrySdk.autoConfigured[IO](
    _.addMeterProviderCustomizer((b, _) => b.registerMetricReader(exporter.metricReader))
  ).use { autoConfigured =>
    val appRoutes: HttpRoutes[IO] = ???
    val routes = appRoutes <+> PrometheusHttpRoutes.routes[IO]

    EmberServerBuilder.default[IO].withHttpApp(routes.orNotFound).build.useForever
  }
}

Instead of launching a dedicated HTTP server, a user can merge exporter routes with the app routes.

[bio-aeon]

I would take it if no one minds.

[alexcardell]

Does this cover pushing to a push gateway as well (e.g. for serverless)? Or is that separate

[bio-aeon]

From my point of view, this issue covers exactly prometheus exporter part from OpenTelemetry spec. And according to spec, prometheus exporter is a pull metric exporter which responds to HTTP requests. That is, this issue doesn't imply the push model. @iRevive maybe you can comment if there is anything planned outside of spec in the future.

[alexcardell]

Ah yes, it has a MUST not

Maybe I can add a contrib module after the basic prometheus is in? We have otel collectors at $work but only configured for traces and not metrics, we do have a push gateway though

[bio-aeon]

I think it makes sense to create a separate issue where this proposal can be discussed :) But by the way you can use prometheus as opentelemetry backend by enabling OTLP receiver, without pushgateway.

[iRevive]

I would be careful keeping non-spec functionality in the main repository. Eventually, the maintenance burden could catch us up.

If it's really needed, you can implement it here: https://github.com/typelevel/otel4s-experimental.