Kafka Streams interceptors instrumented missing CHILD-OF relationship receive -> send

[ppatierno]

I have a simple Kafka Streams API based application doing just the following processing:

builder.stream(this.topicIn, Consumed.with(Serdes.String(), Serdes.String()))
                .mapValues(s -> s.toUpperCase())
                .to(this.topicOut);

When it's instrumented by using the consumer and producer interceptors, I can see two spans: receive and send. I expect the process span missing because it's something I should add somehow in the application when doing the mapValues I guess (which instead happens automatically by using the java agent). The issue I see is that the two spans are completely unrelated, so send is not child of receive. When using interceptors across two different applications (a producer and consumer), the receive is child of send as expected.

My thought is because of the following in the TracingConsumer implementation https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/kafka/kafka-clients/kafka-clients-2.6/library/src/main/java/io/opentelemetry/instrumentation/kafkaclients/TracingConsumerInterceptor.java#L26

It build and finish the span, linking to an existing one (in the case of a send -> receive) but doesn't take into account that the receive could be followed by a send (and maybe a process in the middle). How do we should deal with this from a tracing point of view?

[laurit]

@ppatierno Do you see send and receive in different traces only when you are using kafka library instrumentation and in the same trace when using javaagent? If so then probably this is because we have kafka-streams instrumentation inside javaagent but we don't have a similar library instrumentation for kafka-streams.

[ppatierno]

Do you see send and receive in different traces only when you are using kafka library instrumentation and in the same trace when using javaagent?

Yes, exactly.

If so then probably this is because we have kafka-streams instrumentation inside javaagent but we don't have a similar library instrumentation for kafka-streams.

I think that it matches my description above. The TracingConsumer doesn't expect to have a receive to be the parent of a send, it's always the other way around compared to what happens in a Kafka Streams based application.

[asoldo11]

Hi, are there any plans for this requirement? Or if there's a way to avoid this problem?