toddpalino
commented
5 years ago This is an issue with not being able to talk to the Kafka cluster properly, not burrow specifically. The client is getting an EOF when fetching metadata, which generally means that the server closed the connection for some reason. If you're using client certificates, this could be because of a certificate mismatch, or some other issue. I'd take a look on the server side and see if there is an error message to work with.
burrow 1.1.0 or master (12e681a3a8a61f84f17677996dc3e6a2b79fac41) kafka 1.1.0 sasl_plaintext
[logging] level="debug"
[client-profile.myclient] kafka-version="1.1.0" client-id="burrow-myclient" sasl="saslprofile"
[sasl.saslprofile] username="xxxxx" password="xxxxxx" handshake-first=true
[zookeeper] servers=[ "kf1:2181" ] timeout=6 root-path="/burrow"
[cluster.local] class-name="kafka" servers=[ "kf1:9092","kf2:9092","kf3:9092" ] topic-refresh=60 offset-refresh=30 client-profile="myclient"
[consumer.local] class-name="kafka" cluster="local" zookeeper-path="/local" zookeeper-timeout=30 servers=[ "kf1:9092", "kf2:9092", "kf3:9092" ]
[httpserver.default] address=":8000"
while starting burrow fails with kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
added debug for samana for broker communication:
[sarama] 2018/08/20 16:18:48 client/metadata fetching metadata for all topics from broker kf2:9092 [sarama] 2018/08/20 16:18:48 SASL authentication successful with broker kf2:9092:4 - [0 0 0 0] [sarama] 2018/08/20 16:18:48 Connected to broker at kf2:9092 (unregistered) [sarama] 2018/08/20 16:18:48 client/brokers registered new broker #131368232 at kf3:9092 [sarama] 2018/08/20 16:18:48 client/brokers registered new broker #567700950 at kf2:9092 [sarama] 2018/08/20 16:18:48 client/brokers registered new broker #926183260 at kf1:9092 [sarama] 2018/08/20 16:18:48 client/metadata fetching metadata for all topics from broker kf2:9092 [sarama] 2018/08/20 16:18:48 SASL authentication successful with broker kf2:9092:4 - [0 0 0 0] [sarama] 2018/08/20 16:18:48 Connected to broker at kf2:9092 (registered as #567700950) [sarama] 2018/08/20 16:18:48 SASL authentication successful with broker kf1:9092:4 - [0 0 0 0] [sarama] 2018/08/20 16:18:48 Connected to broker at kf1:9092 (registered as #926183260) [sarama] 2018/08/20 16:18:48 SASL authentication successful with broker kf3:9092:4 - [0 0 0 0] [sarama] 2018/08/20 16:18:48 Connected to broker at kf3:9092 (registered as #131368232) [sarama] 2018/08/20 16:18:48 client/metadata fetching metadata for all topics from broker kf1:9092 [sarama] 2018/08/20 16:18:48 Connected to broker at kf1:9092 (unregistered) [sarama] 2018/08/20 16:18:48 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:48 Closed connection to broker kf1:9092 [sarama] 2018/08/20 16:18:48 client/metadata fetching metadata for all topics from broker kf3:9092 [sarama] 2018/08/20 16:18:48 Connected to broker at kf3:9092 (unregistered) [sarama] 2018/08/20 16:18:48 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:48 Closed connection to broker kf3:9092 [sarama] 2018/08/20 16:18:48 client/metadata fetching metadata for all topics from broker kf2:9092 [sarama] 2018/08/20 16:18:48 Connected to broker at kf2:9092 (unregistered) [sarama] 2018/08/20 16:18:48 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:48 Closed connection to broker kf2:9092 [sarama] 2018/08/20 16:18:48 client/metadata no available broker to send metadata request to [sarama] 2018/08/20 16:18:48 client/brokers resurrecting 3 dead seed brokers [sarama] 2018/08/20 16:18:48 client/metadata fetching metadata for all topics from broker kf1:9092 [sarama] 2018/08/20 16:18:48 Connected to broker at kf1:9092 (unregistered) [sarama] 2018/08/20 16:18:48 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:48 Closed connection to broker kf1:9092 [sarama] 2018/08/20 16:18:48 client/metadata fetching metadata for all topics from broker kf3:9092 [sarama] 2018/08/20 16:18:48 Connected to broker at kf3:9092 (unregistered) [sarama] 2018/08/20 16:18:48 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:48 Closed connection to broker kf3:9092 [sarama] 2018/08/20 16:18:48 client/metadata fetching metadata for all topics from broker kf2:9092 [sarama] 2018/08/20 16:18:48 Connected to broker at kf2:9092 (unregistered) [sarama] 2018/08/20 16:18:48 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:48 Closed connection to broker kf2:9092 [sarama] 2018/08/20 16:18:48 client/metadata no available broker to send metadata request to [sarama] 2018/08/20 16:18:48 client/brokers resurrecting 3 dead seed brokers [sarama] 2018/08/20 16:18:49 client/metadata fetching metadata for all topics from broker kf1:9092 [sarama] 2018/08/20 16:18:49 Connected to broker at kf1:9092 (unregistered) [sarama] 2018/08/20 16:18:49 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:49 Closed connection to broker kf1:9092 [sarama] 2018/08/20 16:18:49 client/metadata fetching metadata for all topics from broker kf3:9092 [sarama] 2018/08/20 16:18:49 Connected to broker at kf3:9092 (unregistered) [sarama] 2018/08/20 16:18:49 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:49 Closed connection to broker kf3:9092 [sarama] 2018/08/20 16:18:49 client/metadata fetching metadata for all topics from broker kf2:9092 [sarama] 2018/08/20 16:18:49 Connected to broker at kf2:9092 (unregistered) [sarama] 2018/08/20 16:18:49 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:49 Closed connection to broker kf2:9092 [sarama] 2018/08/20 16:18:49 client/metadata no available broker to send metadata request to [sarama] 2018/08/20 16:18:49 client/brokers resurrecting 3 dead seed brokers [sarama] 2018/08/20 16:18:49 client/metadata fetching metadata for all topics from broker kf1:9092 [sarama] 2018/08/20 16:18:49 Connected to broker at kf1:9092 (unregistered) [sarama] 2018/08/20 16:18:49 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:49 Closed connection to broker kf1:9092 [sarama] 2018/08/20 16:18:49 client/metadata fetching metadata for all topics from broker kf3:9092 [sarama] 2018/08/20 16:18:49 Connected to broker at kf3:9092 (unregistered) [sarama] 2018/08/20 16:18:49 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:49 Closed connection to broker kf3:9092 [sarama] 2018/08/20 16:18:49 client/metadata fetching metadata for all topics from broker kf2:9092 [sarama] 2018/08/20 16:18:49 Connected to broker at kf2:9092 (unregistered) [sarama] 2018/08/20 16:18:49 client/metadata got error from broker while fetching metadata: EOF [sarama] 2018/08/20 16:18:49 Closed connection to broker kf2:9092 [sarama] 2018/08/20 16:18:49 client/metadata no available broker to send metadata request to [sarama] 2018/08/20 16:18:49 client/brokers resurrecting 3 dead seed brokers {"level":"error","ts":1534781929.483589,"msg":"failed to start client","type":"module","coordinator":"consumer","class":"kafka","name":"local","error":"kafka: client has run out of available brokers to talk to (Is your cluster reachable?)"}
for some reason only first connection to the brokers for collecting metadata was successful, next attempts fails. nothing interesting in kafka logs. can somebody help me with this?