search

spring-cloud / spring-cloud-stream

Framework for building Event-Driven Microservices
http://cloud.spring.io/spring-cloud-stream
Apache License 2.0
1.01k stars 614 forks source link

Force disable the creation of Kafka Admin Client #2914

Closed akoufa closed 8 months ago

akoufa commented 8 months ago

I am using Spring Cloud Stream Binder Kafka to connect to an Azure Event Hub Kafka. I am observing a strange behaviour that Kafka AdminClient is being created despite being disabled by the auto topic creation property. Is there a workaround to force disable the Kafka Admin Client creation?

My configuration looks like this:

spring:
  cloud:
    azure:
      eventhubs:
        connection-string: ${azure.eventhub.connection-string}
    stream:
      kafka:
        binder:
          autoCreateTopics: false
          brokers: ${azure.eventhub.namespace}
          configuration:
            security.protocol: SASL_SSL
            sasl.mechanism: PLAIN
            sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="${azure.eventhub.connection-string}";
          consumer-properties:
            heartbeat.interval.ms: 3000
            session.timeout.ms: 30000
            max.poll.interval.ms: 300000
sobychacko commented 8 months ago

Disabling autoCreateTopic will not prevent AdminClient to be created as the binder needs the admin to perform other broker related tasks. At the moment, you cannot control that behavior in the binder. What is the use case to not have the AdminClient created?

akoufa commented 8 months ago

@sobychacko What are the broker related tasks that are needed for a consuming only application?

sobychacko commented 8 months ago

When the binder establishes the consumer binding, there is an unconditional call to the broker to get the partitions to listen from. For that, we use the AdminClient. See here: https://github.com/spring-cloud/spring-cloud-stream/blob/main/binders/kafka-binder/spring-cloud-stream-binder-kafka-core/src/main/java/org/springframework/cloud/stream/binder/kafka/provisioning/KafkaTopicProvisioner.java#L641

This is triggered from here: https://github.com/spring-cloud/spring-cloud-stream/blob/main/binders/kafka-binder/spring-cloud-stream-binder-kafka/src/main/java/org/springframework/cloud/stream/binder/kafka/KafkaMessageChannelBinder.java#L613