Can't found the trace chain between Armeria service when use opentelemetry as java agent

Describe the bug

Recently I try to use the opentelemetry in our armeria service, but found can't find the trace chain in the jaeger.

for example: spring-boot-tomcat -> spring-boot-jetty -> demo, it will found the trace chain in the jaeger normally,

the result in our service is that: service A call service B , service A and service B is use the build in armeria server( not tomcat and jetty). Only found the service A in the Jaeger, don't found the Service B in the Jaeger.

Steps to reproduce

to make it sample, I try to reproduce it in the armeria example project.

call chain is : spring-boot-jetty -> spring-boot-tomcat -> spring-boot-minimal

spring-boot-minimal(no changes, port is 8080):

@Component
@Validated
@ExceptionHandler(ValidationExceptionHandler.class)
public class HelloAnnotatedService {

    @Get("/")
    public String defaultHello() {
        return "Hello, world! Try sending a GET request to /hello/armeria";
    }

    @Get("/hello/{name}")
    public String hello(
            @Size(min = 3, max = 10, message = "name should have between 3 and 10 characters")
            @Param String name) {
        return String.format("Hello, %s! This message is from Armeria annotated service!", name);
    }
}

how to run spring-boot-minimal

./gradlew clean bootJar
java -javaagent:/Users/user/Downloads/opentelemetry-javaagent.jar \
     -Dotel.resource.attributes=service.name=spring-boot-minimal \
     -Dotel.traces.exporter=jaeger \
     -Dotel.metrics.exporter=none \
     -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
     -jar ./build/libs/spring-boot-minimal.jar

spring-boot-tomcat(change port to 8180) :


@RestController
public class HelloController {
    @RequestMapping(method = RequestMethod.GET, path = "/hello")
    String hello() {
        return "Hello, from tomcat &  "
                + WebClient.of("http://localhost:8080")
                        .get("/hello/minimal")
                        .aggregate()
                        .join()
                        .contentUtf8();
    }
}

how to run spring-boot-tomcat

./gradlew clean bootJar
java -javaagent:/Users/user/Downloads/opentelemetry-javaagent.jar \
     -Dotel.resource.attributes=service.name=spring-boot-tomcat \
     -Dotel.traces.exporter=jaeger \
     -Dotel.metrics.exporter=none \
     -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
     -jar ./build/libs/spring-boot-tomcat.jar

spring-boot-jetty(change port to 8280):


@RestController
public class HelloController {

    @RequestMapping(method = RequestMethod.GET, path = "/hello")
    String hello() {
        return "Hello, from jetty & "
                + WebClient.of("http://localhost:8180")
                        .get("/hello")
                        .aggregate()
                        .join()
                        .contentUtf8();
    }
}

how to run spring-boot-jetty

./gradlew clean bootJar
java -javaagent:/Users/user/Downloads/opentelemetry-javaagent.jar \
     -Dotel.resource.attributes=service.name=spring-boot-jetty \
     -Dotel.traces.exporter=jaeger \
     -Dotel.metrics.exporter=none \
     -Dotel.exporter.otlp.endpoint=http://localhost:4317 \
     -jar ./build/libs/spring-boot-jetty.jar

When I run curl http://localhost:8280/hello , the response is Hello, from jetty & Hello, from tomcat & Hello, minimal! This message is from Armeria annotated service!

Expected behavior

can find the spring-boot-minimal in the below call chain:

Actual behavior

But the call chain is like below, trace id is end in the spring boot minimal.

don't find the spring-boot-minimal:

If I call the spring-boot-minimal curl http://localhost:8080/hello/test, it will work fine and can find the trace in the Jeager.

Javaagent or library instrumentation version

1.31.0

Environment

JDK: corretto-17.0.6 OS: macOS 13.6

Armeria: 1.25.2

Jaeger v1.48.0

Additional context

I am not sure it is the issue of Armeria , or an issue of opentelemetry java agent.

cc @jrhee17

Currently, I believe the issue is when sending requests from (ServerA -> ServerB -> ServerC), traces for ServerC are not collected. I believe this is because spans for an armeria's server aren't collected by default.

Here's a repro of using instrumentation using otel java agent.

https://github.com/jrhee17/my-armeria-otel-sandbox

I've also verified that only CLIENT spans are exported (and there aren't any SERVER spans)

[otel.javaagent 2023-10-19 16:52:35:815 +0900] [armeria-common-worker-kqueue-3-6] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'GET' : 6dffe9b0c1e2820249f084cf024ecdd4 f1da62614b9b3691 CLIENT [tracer: io.opentelemetry.armeria-1.3:1.31.0-alpha] AttributesMap{data={net.sock.peer.port=62660, thread.id=36, user_agent.original=armeria/1.25.2, net.peer.name=127.0.0.1, net.peer.port=62662, thread.name=armeria-common-worker-kqueue-3-5, http.response_content_length=5, http.method=GET, net.protocol.version=1.1, http.url=http://127.0.0.1:62662/, http.status_code=200, net.protocol.name=http}, capacity=128, totalAddedValues=12}
[otel.javaagent 2023-10-19 16:52:35:815 +0900] [armeria-common-worker-kqueue-3-2] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'GET' : b584e74fc839ed5626312725ced45a37 d319c06b66aec765 CLIENT [tracer: io.opentelemetry.armeria-1.3:1.31.0-alpha] AttributesMap{data={net.sock.peer.port=62662, net.sock.peer.addr=127.0.0.1, thread.id=1, thread.name=Test worker, http.response_content_length=5, http.method=GET, net.protocol.version=1.1, http.url=/, http.status_code=200, net.sock.peer.name=127.0.0.1, net.protocol.name=http}, capacity=128, totalAddedValues=11}
[otel.javaagent 2023-10-19 16:52:35:815 +0900] [armeria-common-worker-kqueue-3-4] INFO io.opentelemetry.exporter.logging.LoggingSpanExporter - 'GET' : 7410e6f00b728d40cbe4ad07ff9381c5 3d0f8f4b00543a24 CLIENT [tracer: io.opentelemetry.armeria-1.3:1.31.0-alpha] AttributesMap{data={net.sock.peer.port=62661, thread.id=34, user_agent.original=armeria/1.25.2, net.peer.name=127.0.0.1, net.peer.port=62662, thread.name=armeria-common-worker-kqueue-3-3, http.response_content_length=5, http.method=GET, net.protocol.version=1.1, http.url=http://127.0.0.1:62662/, http.status_code=200, net.protocol.name=http}, capacity=128, totalAddedValues=12}

open-telemetry / opentelemetry-java-instrumentation