Automatic trace continuation

[cykl]

Current TracingKafkaConsumer behaviour is to override Consumer#poll to:

• create very short lived consumer spans before returning all the records to the caller.
• inject span context into record headers to allow trace continuation.

Continuing a trace requires some boilerplate (see https://github.com/PacktPublishing/Mastering-Distributed-Tracing/tree/master/Chapter05 or https://github.com/opentracing-contrib/java-kafka-client/issues/59) and seems a frequent use case.

I believe automatic trace continuation is possible and would provide a better developer experience. Basically, we could subclass ConsumerRecords to create a consumer span when client ask for the next record and finish it on subsequent hasNext invocation.

As trace continuation should be optional, we could set a tracing policy using TracingKafkaConsummerBuilder. Current behaviour could be kept as-is or reimplemented using the same mecanism. Spans would be finished immediately rather than waiting for the next invocation. Start times would differ but I don't know which is better.

Ideally, I had like to be able to decide whether producer span is a parent or only a reference/link to force a new sampling decision. However, it seems only possible with OpenTelemetry not OpenTracing.

I will likely give this idea a try. @malafeev @yurishkuro Any thoughts or experience to share?

[rquinio]

Hi, I'm trying to understand why consumer spans were designed this way (finished on creation). From my naive perspective I'd expect the consumer span to be finished only when offset commit is acknowledged and so it would encompass any processing and there would be no need for automatic continuation... Is this a limitation of Kafka APIs/threading model that #poll and #commit are executed in different threads, so it's no possible to keep track of span+offset status in a ThreadLocal ?

[malafeev]

yes, one issue is that #poll and #commit are executed in different threads, another that #poll returns several messages and for each message new consumer span is created because each message can be from different producer.