Distributed tracing support through Kafka integration

[tak1n]

From what I can tell the Java and NodeJS integration for kafka support distributed tracing:

https://docs.datadoghq.com/tracing/setup_overview/compatibility_requirements/java/#networking-framework-compatibility

Datadog’s Kafka integration works with Kafka version 0.11+, which supports the Header API. This API is used to inject and extract trace context.

https://github.com/DataDog/dd-trace-js/blob/master/packages/datadog-plugin-kafkajs/src/index.js#L64

Is this a wanted feature and doable? If yes I could try to craft a PR with some guidance.

[tak1n]

Not sure how this should work, the operation for consuming is called kafka.consumer.process_batch and from the traces I have in front of me it seems 1 trace represents multiple calls of Consumer#consume.

We are using https://github.com/karafka/karafka and it seems one can decide whether todo batch consuming or single message consuming https://github.com/karafka/karafka/wiki/Consuming-messages#single-message-consuming.

Means distributed tracing in theory could only work in single message consuming mode.

[marcotc]

👋 @tak1n, we have a large effort at Datadog around improving tracing for distributed payloads, Kafka being the most popular system representing such payloads today.

This effort is being championed by the Java team and thus will cascade down to other languages after the groundwork in both the Java tracer and backend/UI have been flushed out.

ddtrace will follow suit when the internal specs are more flushed out and Kafka will definitely be one of the supported use cases.

At this moment, I would not recommend you tackle any work regarding adding such support do ddtrace, as our data modelling and presentation for traces today is insufficient to captured the complexity of all possible Kafka scenarios.

[tak1n]

@marcotc thx, for getting back to me and clarifying it. This sounds great and I'm really excited how it will shape up 😃

[jeffscelza]

Has there been any movement on this?

[felippemr]

@marcotc any news on this? Apologies for the spam.

[tostroski]

This is exactly what I am looking for. Is there any information about where this stands?

[delner]

Hey folks, sorry for the delayed response on this. The Ruby team is spread a bit thin, and other teams at Datadog have been doing quite a bit on this topic; we're trying to get a full picture of how Ruby support should fit in with it.

Current expectation is that we (dd-trace-rb) should implement some headers within our Kafka instrumentation (in this library) to support distributed tracing with Kafka. At the moment, the details of the tech specs still escape me... trying to get more clarity on that from other teams.

If we were to support the implementation of this feature, are there any interested parties that 1) would like to help us develop it, or 2) trial the use of a prototype to validate it? We'd happily lend whatever we can to enable a community contribution here, connect them with the necessary details/people. Let me know if you're interested!

[nvh0412]

Hey @delner , we're heavily using Kafka in our system, particularly in our Ruby applications, would it be possible for you to let me know the status of the tech specs for this Kafka Distributed tracing? eager to assist in implementing and validating the prototype.