Tracing doesn't work for Netty/Micronaut + HTTP/2

[s-af]

Describe the bug

Tracing for server and client http requests doesn't work in combination with Netty/Micronaut + HTTP/2.

Steps to reproduce

Set up simple Micronaut application with netty server. Create controller:

@Controller("/")
public class SimpleController {

    @Get(produces = MediaType.TEXT_PLAIN)
    public String index() {
        return "Hello World";
    }
}

Configure server protocol in application.properties:

micronaut.server.http-version=2.0
micronaut.server.ssl.enabled=true
micronaut.server.ssl.build-self-signed=true

Configure otel to print traces into log:

otel.traces.exporter=logging

Run application and send curl request:

curl.exe https://localhost:8443 -kv --http2
* Host localhost:8443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=localhost
*  start date: May 24 08:50:09 2023 GMT
*  expire date: Dec 31 23:59:59 9999 GMT
*  issuer: CN=localhost
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type ? (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost:8443/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: localhost:8443]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.8.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: localhost:8443
> User-Agent: curl/8.8.0
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< date: Thu, 23 May 2024 08:51:08 GMT
< content-type: text/plain
< content-length: 11
<
Hello World

Expected behavior

Traces appears in console.

When I use http1.1 curl.exe https://localhost:8443 -kv --http1.1 this message appears in logs:

[otel.javaagent 2024-05-23 11:56:57:206 +0300] [default-nioEventLoopGroup-1-4] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'GET' : 279e84e0ccd0d8666543b096a1762b4c 6fede1da0048861a SERVER [tracer: io.opentelemetry.netty-4.1:2.3.0-alpha] AttributesMap{data={network.protocol.version=1.1, user_agent.original=curl/8.8.0, url.scheme=https, thread.name=default-nioEventLoopGroup-1-4, network.peer.port=55652, http.request.method=GET, url.path=/, network.peer.address=0:0:0:0:0:0:0:1, server.address=localhost, client.address=0:0:0:0:0:0:0:1, server.port=8443, thread.id=49, http.response.status_code=200}, capacity=128, totalAddedValues=13}

Actual behavior

Empty traces

Javaagent or library instrumentation version

1.35.0

Environment

JDK: 17 OS: Win10

Additional context

Can share a reproducible application if it helps.

[laurit]

Our netty instrumentation does not support tracing http/2 requests yet.

[trask]

@s-af have you tried the Micronaut instrumentation? https://micronaut-projects.github.io/micronaut-tracing/latest/guide/#opentelemetry

[s-af]

@trask checked server spans with Micronaut Tracing. It works with http/2.

However in our app we intentionally do not use Micronaut Tracing. Would be great if tracing work with Agent.

[vishaltk]

@trask is there any work around to surface http metrics? like throughput & latency for each rest api? I have the following dependencies added in the pom. However, we don't see API level http metrics. Is there any sample/reference micronaut project which we can refer to set up metrics collection with prometheus?

    <dependency>
        <groupId>io.micronaut.micrometer</groupId>
        <artifactId>micronaut-micrometer-core</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micronaut</groupId>
        <artifactId>micronaut-management</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micronaut.micrometer</groupId>
        <artifactId>micronaut-micrometer-registry-prometheus</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>
    <!-- https://mvnrepository.com/artifact/io.micronaut.tracing/micronaut-tracing-opentelemetry-http -->
    <dependency>
        <groupId>io.micronaut.tracing</groupId>
        <artifactId>micronaut-tracing-opentelemetry-http</artifactId>
        <version>6.6.0</version>
    </dependency>

[trask]

@trask is there any work around to surface http metrics?

do you mean specifically Micronaut support in the Java agent? currently no one is working on this, but it is a contribution we would accept

[vishaltk]

Yes @trask , I meant the micronaut support in the java agent. Since netty instrumentation is not supported, I switched to jetty after following this page. After this migration, i could see the http_route in the metric, but that's not showing the actual path of the API. It just says "/*". Has anyone tried building APIs with Jetty on micronaut and succeed surfacing the http metrics?

http_server_request_duration_seconds_bucket{error_type="500",http_request_method="GET",http_response_status_code="500",*http_route="/"**,network_protocol_version="1.1",otel_scope_name="io.opentelemetry.jetty-8.0",otel_scope_version="2.4.0-alpha",url_scheme="http",le="0.005"} 1

[laurit]

@vishaltk miconaut specific instrumentation is needed for better route support

[vishaltk]

@laurit could you please share some more details or a documentation link to achieve micronaut specific instrumentation of jetty?

[laurit]

@vishaltk I'd imagine that it would involve instrumenting micronaut code and inserting a call to update in https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/semconv/http/HttpServerRoute.java