OpenTelemetry HTTP library causes all Micronaut HTTP client requests to fail in Kotlin Coroutines when using the OpenTelemetry Java Agent

[natalie-zamani]

Expected Behavior

We're attempting to use the micronaut.tracing.opentelemetry.http library in conjunction with our Kotlin micronaut app (using coroutines), instrumented with the OpenTelemetry Agent.

We would expect the library to capture additional details on our spans for sever HTTP requests/responses, as well as Micronaut HTTP client requests/responses.

This worked mostly seamlessly for us before we added the OpenTelmetry java agent to our application.

Actual Behaviour

Any interaction with a Micronaut HTTP client fails with the following stack trace:

Cannot invoke "io.opentelemetry.context.Context.makeCurrent()" because "newContext" is null
java.lang.NullPointerException: Cannot invoke "io.opentelemetry.context.Context.makeCurrent()" because "newContext" is null
    at io.micronaut.tracing.opentelemetry.instrument.http.client.OpenTelemetryClientFilter.doFilter(OpenTelemetryClientFilter.java:78)
    at io.micronaut.http.filter.HttpClientFilter.doFilter(HttpClientFilter.java:53)
    at io.micronaut.http.client.netty.DefaultHttpClient.applyFilterToResponsePublisher(DefaultHttpClient.java:1217)
    at io.micronaut.http.client.netty.DefaultHttpClient.exchangeImpl(DefaultHttpClient.java:1078)
    at io.micronaut.http.client.netty.DefaultHttpClient.lambda$exchange$7(DefaultHttpClient.java:753)
    at reactor.core.publisher.FluxSwitchMapNoPrefetch$SwitchMapMain.subscribeInner(FluxSwitchMapNoPrefetch.java:210)
    at reactor.core.publisher.FluxSwitchMapNoPrefetch$SwitchMapMain.onNext(FluxSwitchMapNoPrefetch.java:164)
    at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.withActiveSpan(TracingSubscriber.java:86)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onNext(TracingSubscriber.java:62)
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
    at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.lambda$onNext$1(TracingSubscriber.java:62)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.withActiveSpan(TracingSubscriber.java:86)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onNext(TracingSubscriber.java:62)
    at io.micronaut.core.async.publisher.Publishers$JustPublisher$1.request(Publishers.java:563)
    at reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164)
    at reactor.core.publisher.FluxSwitchMapNoPrefetch$SwitchMapMain.onSubscribe(FluxSwitchMapNoPrefetch.java:147)
    at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onSubscribe(ReactorSubscriber.java:50)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.lambda$onSubscribe$0(TracingSubscriber.java:57)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.withActiveSpan(TracingSubscriber.java:86)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onSubscribe(TracingSubscriber.java:57)
    at reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:92)
    at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onSubscribe(ReactorSubscriber.java:50)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.lambda$onSubscribe$0(TracingSubscriber.java:57)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.withActiveSpan(TracingSubscriber.java:86)
    at io.opentelemetry.javaagent.shaded.instrumentation.reactor.v3_1.TracingSubscriber.onSubscribe(TracingSubscriber.java:57)
    at io.micronaut.core.async.publisher.Publishers$JustPublisher.subscribe(Publishers.java:553)
    at reactor.core.publisher.FluxSource.subscribe(FluxSource.java:67)
    at reactor.core.publisher.Flux.subscribe(Flux.java:8660)
    at reactor.core.publisher.Flux.blockFirst(Flux.java:2700)
    at io.micronaut.http.client.netty.DefaultHttpClient$1.exchange(DefaultHttpClient.java:499)
    at io.micronaut.http.client.BlockingHttpClient.exchange(BlockingHttpClient.java:77)
    at io.micronaut.http.client.BlockingHttpClient.exchange(BlockingHttpClient.java:91)

This is reproducible in tests in our application. We're using the latest Otel java agent version as well.

Steps To Reproduce

No response

Environment Information

• JDK Version: 17

Example Application

No response

Version

3.9.1

[graemerocher]

could you provide steps to reproduce?

[BrianEstrada]

I'm having the same issue, in order to reproduce you just need to have the opentelemetry agent attached to the server and the following dependencies

    // Tracing
    implementation("io.micronaut:micronaut-tracing")
    implementation("io.micronaut.tracing:micronaut-tracing-opentelemetry-http")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp")

[scprek]

Not sure if this is relevant, but I've been updating docs in a few repos as things were not clear related to logback MDC and Kotlin coroutine support https://github.com/open-telemetry/opentelemetry-java/pull/5799