OpenTelemetry Metrics Investigation

[fmhwong]

• Migrate existing sample applications from MicroProfile Metrics to OpenTelemetry Metrics.

[fmhwong]

Presented metrics comparison in a MicroProfile Telemetry call on 8/30/2023

[donbourne]

@fmhwong / @Channyboy, please provide your thoughts on the following question...

Goals

1. applications must be able to use the OTel metrics API directly to create/update metrics
2. Liberty runtime must be able to send runtime and application metrics to OTLP receivers
3. (???) Liberty runtime must be able to provide a Prometheus "/metrics" endpoint for runtime and application metrics. The "/metric" output names/labels must match what you would see if you looked at metrics from Prometheus received from OTel Collector (Liberty -> OTel Collector -> Prometheus)
4. application / runtime metrics must be able to share OTel Context with OTel traces and logs
5. metric names/labels must follow OTel semantic conventions that are "stable" (http / jvm)
6. metric names/labels should follow OTel semantic conventions that currently exist but that aren't stable unless there's a good reason not to (which we would probably want to raise with OTel community).

Questions Q1. There are multiple ways that we could get our Liberty runtime and applications to use OTel metrics. What are the pros/cons of each of the following options? Which option seems best?:

1. Send metrics to MP Metrics and use the OTLP meter registry with Micrometer to be able to forward to an OTLP receiver. Let applications use OTel API or MP Metrics API to create/update metrics.
2. Send Liberty runtime metrics to OTel API using the "MappingTable" approach for metrics (and directly using OTel Timer API for Timer metrics). Let applications use OTel API to create/update metrics.
3. Update metrics instrumentation points in Liberty to use Observations. Provide Observation handlers to route metrics to MP Metrics or OTel (or potentially others in the future). Let applications use OTel API to create/update metrics. Possibly let applications add Observation handlers.

@Emily-Jiang, FYI

[fmhwong]

I tested out the OpenTelemetry JVM Runtime Metrics for Java 8 on multiple versions of Java and recorded the results at https://github.com/eclipse-openj9/openj9/issues/18952

[Channyboy]

1. Pro:
   • Quick?

Con:

• Need to start mpMetrics to have the Open Telemetry support
• User would have to deal with the MP Metrics' Configuration properties to configure the buckets for Timers and Histograms before starting server.

Other:

• Any changes in the metrics implementation means we would need to cut new releases for SmallRye metrics. Any problems with micrometer means we'd have to wait for micrometer.
• Unsure if the context can be shared with the rest of the Open telemetry feature functionality? (tracing + logging)?

• Unsure If we're asking them to BYO or we will provide under the covers for Open Liberty? (most likely we provide)

  2. Pro:

• Decoupled from mpMetrics

Con:

• Parallel implementation of code that's similar (w.r.t. the use of mapping tables)?

  3. Pro:

• Decoupled from mpMetrics

Con: Other:

• Assuming Open Telemetry is its own feature, We'd expect both features to get liberty metrics sent to each other regardless of the observation API Implementation, since that is what the user wanted to enable (i.e. mpMetrics and Open Telemetry feature enabled).