search

spring-cloud / spring-cloud-stream-binder-kafka

Spring Cloud Stream binders for Apache Kafka and Kafka Streams
Apache License 2.0
331 stars 301 forks source link

Configure bootstrap servers instead of brokers and zookeeper #180

Closed michael-barker closed 7 years ago

michael-barker commented 7 years ago

It looks like a while back the Kafka consumer gained the ability to not have a dependency on zookeeper allowing you to only configure the brokers. I'm guessing this is what spring-kafka uses. Are there any plans to support this?

garyrussell commented 7 years ago

Yes; spring-kafka uses the new pure java client, which does not need access to zookeeper.

Unfortunately, however, the client does not have a topic provisioning capability.

The 0.11 client added the KafkaAdminClient (KIP-117) which allows the creation of topics, but, unfortunately, it doesn't currently allow modifying (repartitioning) topics; which the binder needs to do if the partitionCount is higher than the currently provisioned topic supports.

From the KIP:

Future Work

We would like to add more functionality to the AdminClient as soon as it becomes available on the brokers. For example, we would like to add a way of altering topics that already exist. We would also like to add management APIs for ACLs, or the functionality of GetOffsetShell.

(My emphasis) So, hopefully, in some future version, we will be able to remove the need for zookeeper configuration.

michael-barker commented 7 years ago

Thank you for the detailed explanation. That more than answers my question.