Documentation clarification for [cluster.mycluster] section configuration

[Daniel314]

Would it be possible to add a note to the documentation about cluster names in the config file?

I found out the hard way that if the cluster name has upper-case characters in it, it breaks things? (Specifically, the burrow program will report "unknown cluster" (in the addConsumerOffset() routine) for all consumer offset updates with a cluster name of "CMU" or "Cmu". It works fine with a cluster name of "cmu")

Here is an example log error message when the cluster name has an upper-case letter in it: {"level":"warn","ts":1517431581.178074,"msg":"unknown cluster","type":"module","coordinator":"storage","class":"inmemory","name":"default","worker":11,"cluster":"Cmu","consumer":"test","topic":"LSpipeline","partition":58,"topic_partition_count":0,"offset":1827590082,"timestamp":1517431580904,"owner":"","request":"StorageSetConsumerOffset"}

Thank you,

- Daniel

[toddpalino]

Can you share your configuration, please? Everything should be case-sensitive (which, ultimately, should be the default assumption for any system unless otherwise specified)

[Daniel314]

Hello Todd,

I've attached three configurations to this e-mail. Two of them work, and one does not (the filenames indicate which is which). Does this give you what you were looking for?

Thank you,

- Daniel

#

Burrow project: https://github.com/linkedin/Burrow

#

[zookeeper] servers=[ "kafka1.coloradomesa.edu:2181" ] timeout=6 root-path="/burrow"

NOTE: cluster names can NOT have upper-case letters in them!

If you do, you'll likely see "unknown cluster" errors and no offset data.

[cluster.CMU] class-name="kafka" servers=[ "kafka1:9092", "kafka2:9092", "kafka3:9092" ] topic-refresh=60 offset-refresh=20

[consumer.kafka9plusClient] class-name="kafka" cluster="CMU" offsets-topic="__consumer_offsets" start-latest=true servers=[ "kafka1:9092", "kafka2:9092", "kafka3:9092" ] group-blacklist="^(console-consumer-|python-kafka-consumer-).*$" group-whitelist=""

[consumer.kafkaZKClient] class-name="kafka_zk" cluster="CMU" servers=[ "kafka1:2181", "kafka2:2181", "kafka3:2181" ] zookeeper-path="/elk" zookeeper-timeout=30 group-blacklist="^(console-consumer-|python-kafka-consumer-).*$" group-whitelist=""

[httpserver.default] address=":8001"

[logging]

level="debug"

#

Burrow project: https://github.com/linkedin/Burrow

#

[zookeeper] servers=[ "kafka1.coloradomesa.edu:2181" ] timeout=6 root-path="/burrow"

NOTE: cluster names can NOT have upper-case letters in them!

If you do, you'll likely see "unknown cluster" errors and no offset data.

[cluster.cmu] class-name="kafka" servers=[ "kafka1:9092", "kafka2:9092", "kafka3:9092" ] topic-refresh=60 offset-refresh=20

[consumer.kafka9plusClient] class-name="kafka" cluster="cmu" offsets-topic="__consumer_offsets" start-latest=true servers=[ "kafka1:9092", "kafka2:9092", "kafka3:9092" ] group-blacklist="^(console-consumer-|python-kafka-consumer-).*$" group-whitelist=""

[consumer.kafkaZKClient] class-name="kafka_zk" cluster="cmu" servers=[ "kafka1:2181", "kafka2:2181", "kafka3:2181" ] zookeeper-path="/elk" zookeeper-timeout=30 group-blacklist="^(console-consumer-|python-kafka-consumer-).*$" group-whitelist=""

[httpserver.default] address=":8000"

[logging]

level="debug"

#

Burrow project: https://github.com/linkedin/Burrow

#

[zookeeper] servers=[ "kafka1.coloradomesa.edu:2181" ] timeout=6 root-path="/burrow"

NOTE: cluster names can NOT have upper-case letters in them!

If you do, you'll likely see "unknown cluster" errors and no offset data.

[cluster.cmutest] class-name="kafka" servers=[ "kafka1:9092", "kafka2:9092", "kafka3:9092" ] topic-refresh=60 offset-refresh=20

[consumer.kafka9plusClient] class-name="kafka" cluster="cmutest" offsets-topic="__consumer_offsets" start-latest=true servers=[ "kafka1:9092", "kafka2:9092", "kafka3:9092" ] group-blacklist="^(console-consumer-|python-kafka-consumer-).*$" group-whitelist=""

[consumer.kafkaZKClient] class-name="kafka_zk" cluster="cmutest" servers=[ "kafka1:2181", "kafka2:2181", "kafka3:2181" ] zookeeper-path="/elk" zookeeper-timeout=30 group-blacklist="^(console-consumer-|python-kafka-consumer-).*$" group-whitelist=""

[httpserver.default] address=":8001"

[logging]

level="debug"

[joelp172]

I can confirm that i was hit by this problem. I used [clusterA] and got the error message "unknown cluster" in the logs.

Making the cluster name all lower case resolves the issue