A consumer thats committing offsets not listed

[sathyakumarseshachalam]

This is more a question rather than an issue :

Just getting started with burrow. I created a topic with 6 partitions and 3 replication factors. And published messages to the two of the partitions via sarama kaka-console-producer.

Then consumed those messages using kafka tool's kaka-console-consumer, I am passing a consumer config that has below entries

group.id=MyBurrowConsumer
offset.storage=kafka
auto.commit.offset = true

I was able to verify that offsets are committed, by describing the consumer group with kafka-consumer-groups.sh

GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
MyBurrowConsumer, burrow_test, 0, 10, 10, 0, none
MyBurrowConsumer, burrow_test, 1, 10, 10, 0, none
MyBurrowConsumer, burrow_test, 2, 0, 0, 0, none
MyBurrowConsumer, burrow_test, 3, 0, 0, 0, none
MyBurrowConsumer, burrow_test, 4, 0, 0, 0, none
MyBurrowConsumer, burrow_test, 5, 0, 0, 0, none

However the HTTP end points does not seem to list MyBurrowConsumer with the end point /v2/kafka/alpha/consumer

So my question is what would it take to get MyBurrowConsumer to be listed

[sathyakumarseshachalam]

Even more weirder is the fact that amongst the consumer listed - Lets say AConsumerGroup. This is a consumer for which /v2/kafka/alpha/AConsumerGroup/lag returns a value. But kafka.tools.ConsumerOffsetChecker complains that NoNode for /consumers/AConsumerGroup/owners. So my question how does burrow arrive at the list of consumer groups. And is it possible to handle a case where in the same cluster, there is a section of consumers that commit offset to kafka and a section that commits the offset to Zk.

[sathyakumarseshachalam]

I had a typo in my consumer config verified that offsets.storage=true makes /v2/kafka/alpha/consumer list MyBurrowConsumer. However my question still stands.

How to make burrow also list consumers who have their offset committing to Zk. Adding

[kafka "alpha"]
...
...
zookeeper-offsets=true

does not seem to make a difference.

[obaidsalikeen]

I am experiencing the same issue. Burrow is not able to list Storm consumers in separate ZK cluster. Any luck in getting it to work?

[sathyakumarseshachalam]

@obaidsalikeen : Nope, My consumers are all committing to Kafka, so is not a blocker for me.

[obaidsalikeen]

@sathyakumarseshachalam : Thanks for your reply. I have few more questions :) By 'committing to Kafka' you mean committing to Kafka Zookeeper? Which version of Kafka you are using, i mean are you using __consumer_offset? How is your experience so far with Burrow, have you tried https://github.com/quantifind/KafkaOffsetMonitor, if so is Burrow still a good option for monitoring consumers (use in production)? Thank

[sathyakumarseshachalam]

@obaidsalikeen I mean Kafka itself - __consumer_offset topic. Still evaluation Burrow, so haven't been able to form an opinion.

[toddpalino]

@sathyakumarseshachalam To your original question, as to how Burrow lists consumer groups, it only knows about a consumer group when it commits offsets. Burrow does not use any other means of getting a full group list. If a consumer commits an offset, the group exists. If there is no offset committed, then as far as Burrow is concerned the group does not exist. If you have enabled Zookeeper offset checking for Kafka, Burrow also enumerates the /consumers tree in ZK for groups, but it will only evaluate groups there that have a /consumers//offsets tree.

@obaidsalikeen Storm consumers are a bit different here. Can you open up a separate issue with your Burrow config and what you're seeing?

[sathyakumarseshachalam]

Closing this issue, as there is enough clarity