Poseidon cluster stops consuming if one of the brokers goes away

[Ben-M]

I was experimenting with different failure conditions and found that if I shut down one of the Kafka brokers poseidon_cluster would often fail to rebalance, and did not get messages from some partitions. Am I missing something, or does poseidon_cluster only support rebalancing when consumers join/leave the consumer group? More details:

• I ran zookeeper and 5 Kafka brokers locally (more or less as described here).
• I created a topic with 3 partitions and a replication factor of 3: bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic 3
• I used kafkacat to produce a stream of messages at a reasonable pace: ping -i 0.1 localhost|kafkacat -P -b 127.0.0.1:9093 -t 3. I didn't use kafka-console-producer.sh as it only produces to a single partition
• I set up poseidon_cluster to consume from the topics and output the message and partition to the console.
• I killed one of the brokers using an interrupt signal. If poseidon_cluster was still consuming nicely then I brought it back online and tried again.

[kssminus]

same here.. +1

[Ben-M]

This might be connected to https://github.com/bpot/poseidon/issues/115