OTLP registry custom header support

[stse]

Please describe the feature request. Provide configuration properties to provide custom headers e.g. API-Key which are sent whenever metrics are posted to the OTLP endpoint.

Having something like this

management:
  otlp:
    metrics:
      export:
        step: 1s
        url: https://otlp.eu01.nr-data.net/v1/metrics
        headers:
          API-Key: otlp-endpoint-api-key

so that these headers are sent with each request may be this way

var builder = this.httpSender.post(this.config.url());
for (final var header : this.config.headers()) {
  builder = builder.withHeader(header.key(), header.value());
}
builder.withContent("application/x-protobuf", request.toByteArray()) .send();

see https://github.com/micrometer-metrics/micrometer/blob/029a887eb86476036d28527ebda890289161623a/implementations/micrometer-registry-otlp/src/main/java/io/micrometer/registry/otlp/OtlpMeterRegistry.java#L108

Another solution would be to use the Open Telemetry SDK and allow to provide custom MetricExporter beans. This works very well already for tracing. Here I can provide custom SpanExporter beans.

Rationale Having direct OTLP support is great but in order to connect with some OTLP endpoints, e.g. NewRelics OTLP endpoints a header containing an API key has to be sent with each request.

[shakuzen]

Thank you for opening the issue and describing the use case. With other MeterRegistry implementations, such a need would either be a first class concept (API key) and supported in the Config, or you would be able to customize the HttpSender that is used to achieve the same thing. Unfortunately neither of those are available for OtlpMeterRegistry right now. Generic custom header support sounds a bit too generic to me, but maybe it makes sense. In the initial implementation we intentionally did not expose the HttpSender in the API because OTLP can be sent more ways than HTTP. One option is to introduce a Builder for the registry that allows optionally customizing the HttpSender. I'll discuss with the team.

[stse]

@shakuzen Thanks for your response. Actually I really like the integration of OTLP in Micrometer's tracing implementation, because all abstractions introduced by Open Telemetry and its SDK, e.g. the Exporters are still available. Similar bridge as for the Tracer would be nice because then the Open Telemetry objects can be shared and all exporters that provided by Open Telemetry are available. Not only the http exporter.

[shakuzen]

The current implementation does not prevent us from supporting more than the HTTP transport. The micrometer-registry-otlp module does not depend on the OTel SDK. It depends only on the OTLP protobuf schema. This is by design, because it is not a bridge to the OTel SDK, whereas micrometer-tracing-bridge-otel is. micrometer-registry-otlp is a MeterRegistry implementation that allows publishing OTLP format metrics for users that aren't using OTel SDK. We should be able to publish OTLP format without depending on the OTel SDK.

Since the context here is metrics, what other metrics exporter would you like to see supported? OTLP format metrics over GRPC? Is there anything else? We already have a separate Prometheus registry for Prometheus/OpenMetrics format. MeterRegistry is the abstraction in Micrometer for exporting metrics.

[stse]

Usually I work with Spring Boot and I really like Open Telemetry because it allows me to decouple the tooling to present and evaluate metrics, logs and traces from the tooling to capture metrics, logs and traces. When Spring announces that it will support OTLP with Spring 6 I was really hyped, because I'm not a fan of the Open Telemetry Java Agent which sometimes lags behind in it's framework support and also requires additional steps in the deployment pipeline. So I hoped that I can configure Open Telemetry with Spring Boot and then metrics, logs and traces are published by spring automatically. At the moment I have to configure Open Telemetry for metrics, logs and traces separately. That is the root of my request. So yes it is sufficient for my use case to have a http and/or grpc exporter, but I hoped that I could reuse some of my Open Telemetry knowledge which worked for tracer bridge but not for the meter registry, e.g. how to configure the open telemetry resource attributes service.name, service.namespace, service.instance.id. I was surprised that two different approaches to implement metrics and traces were used. I would have expected that either the Open Telemetry SDK is bridged or that traces as well metrics only rely on the OTLP protocol specification.

[shakuzen]

There are historical reasons why there are different approaches. Micrometer (this repo) predates OpenTelemetry and there were explicit requests from users to have an option for producing OTLP that does not pass through the OpenTelemetry SDK. Micrometer Tracing is largely Spring Cloud Sleuth minus the Spring bits. Hence, there are different approaches that make sense given their historical context. I'm sorry to hear it is causing friction to use them together now. That is something we're interested in improving, but I think we need to move that discussion from this issue as it's not directly related IMO.

On the request here, it looks like it would be reasonable for the OtlpMeterRegistry to support a configuration option for additional headers in a similar way as specified here, with OTEL_EXPORTER_OTLP_HEADERS or OTEL_EXPORTER_OTLP_METRICS_HEADERS. We already have similar support for OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME.

[grepory]

Hi! Was redirected here from Slack, and I wanted to say a couple of things about my particular use cases.

First, I love Micrometer. :) I find it easy to work with, the Spring integration is very easy to use. It's a user experience I've come to appreciate. I've already instrumented some of the code I maintain for personal use for Prometheus, but I'm migrating to OTEL, and I wanted to use Micrometer's OTLP registry to make the transition easier. I don't want to go the Prometheus receiver route and use the otel collector in a separate process if I don't have to. I'd like to just rip Prometheus out of everything.

Now to other use cases I have: I have many services that are instrumented with Micrometer and use the AppOptics integration. Solarwinds' new Observability product (that's eventually going to replace AppOptics) has a public OTEL collector that its integrations use. I'd be able to ease a lot of the migration effort of not only myself, but also other AppOptics+Micrometer users if the registry could simply be swapped out, instead of ripping out all of my instrumentation.

The public collector endpoints requires bearer-token authentication. Injecting the authorization header would be easier if the OtlpMeterRegistry's constructor accepting an HttpSender were public (or protected with a subclass extending Otlp specifically for Solarwinds, to which I'm also amenable). I know that there's concern about supporting other transports, but even if support for gRPC is introduced, you'll still want to support HTTP. gRPC supports bearer-token auth as well.

I'd be happy to help out with this. I've already started doing some of the work for my own personal use.

[shakuzen]

Thank you for sharing your use case and explaining the need for this. We are planning to do this work for the upcoming 1.11 release, with the approach mentioned in my previous comment, unless you have some feedback on that approach not working well for your use case(s).

On the request here, it looks like it would be reasonable for the OtlpMeterRegistry to support a configuration option for additional headers in a similar way as specified here, with OTEL_EXPORTER_OTLP_HEADERS or OTEL_EXPORTER_OTLP_METRICS_HEADERS. We already have similar support for OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME.

[grepory]

This is great. Thank you!

[shakuzen]

I've added a new method to OtlpConfig called headers for configuring additional headers to be sent when publishing metrics from the OtlpMeterRegistry. They can also be configured via the environment variables OTEL_EXPORTER_OTLP_HEADERS or OTEL_EXPORTER_OTLP_METRICS_HEADERS. See the JavaDoc on the new headers method for more details. This is now available in 1.11.0-SNAPSHOT and will be released in the upcoming milestone 1.11.0-M1. Let us know if there is any feedback or issues with it.

[cboudereau]

Hi @shakuzen . It seems that the 1.11.0 milestone is complete. I know that the due date is on May but is it possible to publish the release ?

[shakuzen]

The 1.11.0 release will happen in May as planned. There are multiple issues we need to address before it is ready for GA. I will work on adding those issues to the milestone. We publish feature (minor version) releases on a regular time-based schedule with several milestone releases before GA to get feedback on new features and bigger changes before releasing them in a GA version. This helps identify fundamental issues that would be hard to address after releasing something in a GA version because we do not want to break backward compatibility. We would generally only consider publishing patch releases off-schedule. I'm sorry this means you'll need to wait a bit longer to get this feature in a GA version, but please do give it a try in the latest milestone version (1.11.0-RC1) and let us know if there is feedback on it.