connect to kafka cluster using zookeeper (without knowing the actual kafka cluster)

[timfeirg]

I use zookeeper to manage kafka cluster, but using pykafka one should always initiate a pykafka.cluster.Cluster instance using the kafka cluster hosts, is there anyway to connect to kafka cluster via zookeeper cluster, so that the consumer doesn't care about the actual kafka hosts?

typical java code to do the job:

public static void main(String[] args) {
        Properties props = new Properties();
        props.put("zookeeper.connect", "xxxx:2181,xxxx:2181,xxxx:2181/kafka");
        props.put("zookeeper.session.timeout.ms", "100000");
        props.put("group.id", "test_kakfa_1234");
        props.put("serializer.class", "kafka.serializer.StringEncoder");
​
        ConsumerConfig config = new ConsumerConfig(props);
​
        ConsumerConnector consumer = kafka.consumer.Consumer.createJavaConsumerConnector(config);
​
        Map<String, Integer> map = new HashMap<>();
        map.put("app_action", 1);
​
        StringDecoder keyDecoder = new StringDecoder(new VerifiableProperties());
        StringDecoder valueDecoder = new StringDecoder(new VerifiableProperties());
​
        Map<String, List<KafkaStream<String, String>>> consumerMap =
                consumer.createMessageStreams(map, keyDecoder, valueDecoder);
        KafkaStream<String, String> stream = consumerMap.get("app_action").get(0);
        ConsumerIterator<String, String> it = stream.iterator();
        while (it.hasNext())
            System.out.println(it.next().message());
    }

[emmettbutler]

Hi @timfeirg, thanks for filing this. Pykafka doesn't support this at this time, unfortunately. It does seem like a worthwhile feature, since I don't think kafka is ditching zookeeper anytime soon.

[emmettbutler]

I've taken a shot at this in #364. Let me know if it satisfies your use case @timfeirg.