search

spring-projects / spring-kafka

Provides Familiar Spring Abstractions for Apache Kafka
https://projects.spring.io/spring-kafka
Apache License 2.0
2.16k stars 1.54k forks source link

Kafka Streams tracing no longer working after upgrading to Spring Boot 3 + Micrometer Tracing #2635

Open j-tim opened 1 year ago

j-tim commented 1 year ago

Context

At the moment I'm migrating some Spring Boot 2.7.x, Spring for Apache Kafka applications to Spring Boot 3.

I followed the migration guide and Spring Kafka documentation regarding migrating from Spring Cloud Sleuth to Micrometer Tracing

Distributed tracing works fine for:

But for Kafka streams application (using the KafkaStreamBrancher in my minimal, reproducible, sample) the tracing is no longer working after the upgrade.

In what version(s) of Spring for Apache Kafka are you seeing this issue?

Spring for Apache Kafka: 3.0.4

Migrating from:

to:

Describe the bug

For the Kafka streams application the tracing is no longer working after the upgrade.

Spring Boot 2.7.9, Spring Kafka and Spring Cloud Sleuth:

Spring Boot 3.0.4, Spring Kafka and Micrometer Tracing (OTEL)

Expected behavior

I expect the traces of my Kafka Streams application are reported in this case to Zipkin.

Sample to reproduce the issue

I put some effort in creating a GitHub repository with a minimal, reproducible, sample to reproduce the issue. The README of the sample project contains all the step to reproduce the issue.

Branches in the example repository:

Did I overlook something in the documentation to enable tracing for my Kafka Streams application? Or is tracing for Kafka Stream applications not supported yet using Spring for Apache Kafka and Micrometer Tracing?

garyrussell commented 1 year ago

Spring is not involved with instrumenting Kafka Streams at all - Spring is only used to set up the stream; there is no runtime involvement.

My (limited) understanding is that Sleuth uses brave to instrument the streams producer/consumer.

I don't know if it's practical for this project to replace the sleuth functionality for Streams - I'll have to defer that to the Micrometer team.

cc/ @marcingrzejszczak

marcingrzejszczak commented 1 year ago

If there's an option for Spring Kafka to instrument Kafka Streams by any means then that would be great (although I don't know if that's feasible?).

garyrussell commented 1 year ago

How does Sleuth do it?

marcingrzejszczak commented 1 year ago

As you said - it does it via Brave's components (https://github.com/spring-cloud/spring-cloud-sleuth/blob/c795675cf93c666c3c13f9bbb9eadbde1f1aa775/spring-cloud-sleuth-autoconfigure/src/main/java/org/springframework/cloud/sleuth/autoconfig/brave/instrument/messaging/BraveKafkaStreamsAutoConfiguration.java).

Here you can see how they exactly do it https://github.com/openzipkin/brave/tree/master/instrumentation/kafka-streams

garyrussell commented 1 year ago

I think all we can do here is explain how to set the KafkaClientSupplier on the StreamsBuilderFactoryBean, as is done by Sleuth here:

https://github.com/spring-cloud/spring-cloud-sleuth/blob/c795675cf93c666c3c13f9bbb9eadbde1f1aa775/spring-cloud-sleuth-autoconfigure/src/main/java/org/springframework/cloud/sleuth/autoconfig/brave/instrument/messaging/BraveKafkaStreamsAutoConfiguration.java#L106

marcingrzejszczak commented 1 year ago

Not really cause that's just delegating to what brave does. What if someone uses open telemetry or another tracer?

garyrussell commented 1 year ago

What does Sleuth do in those cases?

The only extension point that Kafka Streams provides is the client supplier; so each library needs to provide its own implementation to decorate the producer/consumer; there is nothing else that this project can do.

marcingrzejszczak commented 1 year ago

Sleuth only supports the dedicated Brave instrumentation.

j-tim commented 1 year ago

Thanks, Gary and Marcin for your responses. I understand just delegating to brave is not the way to go.

Based on your suggestions I try to implement a custom Kafka client supplier. To set the custom KafkaClientSupplier on the StreamsBuilderFactoryBean:

@Bean
public StreamsBuilderFactoryBeanCustomizer streamsBuilderFactoryBeanCustomizer() {
   return factoryBean -> factoryBean.setClientSupplier(new TracingKafkaClientSupplier());
}

The TracingKafkaClientSupplier extends the DefaultKafkaClientSupplier.

Can you give me some hints on how to implement this custom supplier? Should this custom supplier use the ObservationRegistry of Micrometer? Or should I directly use the OpenTelemetry API like in this example: https://github.com/ppatierno/kafka-opentelemetry/blob/main/kafka-streams-wrap/src/main/java/io/ppatierno/kafka/opentelemetry/Streams.java#L42

Would it not make more sense if Spring or Micrometer will provide a 'Tracing Kafka client supplier' that will be auto-configured or registered when observation is enabled like how it's done for the KafkaTemplate and KafkaListenerContainer?

garyrussell commented 1 year ago

Unfortunately, Kafka Streams is a black box; Spring has no way to instrument the underlying producers/consumers that are created. Nor does it have access to scope of a consume or produce operation.

So there is no way to provide the functionality that Spring can provide where we have control over the consumption and production of records.

The instrumentation would have to be done internally by the kafka-clients.

artembilan commented 1 year ago

See this one as an alternative solution: https://github.com/openzipkin/brave/blob/2ef59ea0e687d51e445bb5144c27e373fa73b7d7/instrumentation/kafka-clients/src/main/java/brave/kafka/clients/TracingProducer.java.

It feels like a dedicated library as a kafka-micrometer should emerge instead where an Observation wrapping and handling can be implemented down to that ObservationKafkaClientSupplier. Something similar to what we have now for JDBC: https://jdbc-observations.github.io/datasource-micrometer/docs/current/docs/html/#getting-started-installation.

marcingrzejszczak commented 1 year ago

Ok I see so this has to be done on the kafka level. We already have metrics for Kafka (https://github.com/micrometer-metrics/micrometer/tree/main/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/kafka) so we would need to add observability support, right?

artembilan commented 1 year ago

Sounds legit, @marcingrzejszczak . 👍

marcingrzejszczak commented 1 year ago

Added an issue in Micrometer - https://github.com/micrometer-metrics/micrometer/issues/3713 I guess we can close this in favor of that one?

artembilan commented 1 year ago

I think we will keep this one as a documentation type to mention in the future how to instrument Kafka Streams with an observability when it is ready on a common library.