search

zendesk / ruby-kafka

A Ruby client library for Apache Kafka
http://www.rubydoc.info/gems/ruby-kafka
Apache License 2.0
1.28k stars 337 forks source link

Connecting to Kafka via Zookeeper #274

Closed whithajess closed 8 years ago

whithajess commented 8 years ago

Doesn't look this is currently possible although maybe I am missing something? Is this going to make its way on to the RoadMap?

From what I understand you want to be connecting to Zookeeper not directly to Kafka.

Maybe you could shed some light @dasch?

dasch commented 8 years ago

I think that advice is out of date – as of Kafka 0.9, there's no direct communication between Kafka clients and Zookeeper. You should connect directly to the brokers.

whithajess commented 8 years ago

Im not saying it is necessarily up to date but According to the Kafka 0.10.0/0.9.0.X Documentation:

Creating a topic -

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Sending some messages -

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

Start a Consumer -

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

My experience with other Kafka integrations follow this pattern (Producer to Kafka and Consumer from Zookeeper) see:

https://github.com/SOHU-Co/kafka-node https://github.com/joekiller/jruby-kafka

If you have any references to why this should not be the case I would love to check them out.

tcrayford commented 8 years ago

@withajees the "new" Java producer and consumer don't talk through zookeeper as of 0.9. In fact, as of 0.9, you can use Kafka without requiring a zookeeper connection for most things. Topic creation needs zk still right now, and some of the old scripts and clients still rely on zk, but it is not needed anymore at all.

whithajess commented 8 years ago

Hey Guys, thanks for the info. I also asked on the Kafka user mailing list and they confirmed:

... zookeeper was used to store information about topics, consumers, and offsets (version 0.8.-ish). In newer versions of kafka (0.9. and 0.10.*) those values are instead written to internal topics for Kafka to manage consumers, topics, and offsets.

and that

So much (possibly all?) of the functionality that zookeeper provided is being deprecated in favor of storing that info in kafka.