Kafka Fails when more than 1 zookeeper instances are up

[csarora]

Kafka fails with below error when i have more than 1 replica of zookeeper up. Currently I have zookeeper replica as 3 and kafka also 3. Kafka keeps on going in crashloopbackoff. Error below Note * Works fine when one one zookeeper replica is provided in connect. i.e. when only one zookeeper replica.

Please help.

[root@devtricorder69-master-01 ~]# kubectl get po -l comp-group=kafka
NAME          READY     STATUS             RESTARTS   AGE
kafka-0       0/1       CrashLoopBackOff   6          8m
zookeeper-0   1/1       Running            0          1h
zookeeper-1   1/1       Running            0          1h
zookeeper-2   1/1       Running            0          1h

[2018-02-16 12:32:43,864] INFO Socket connection established to 10.32.0.110/10.32.0.110:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-02-16 12:32:43,866] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
..

2018-02-16 12:32:49,027] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server '10.32.0.108:2181,10.32.0.110:2181,10.32.0.111:2181' with timeout of 9000 ms
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
    at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:106)
    at kafka.utils.ZkUtils$.apply(ZkUtils.scala:88)
    at kafka.server.KafkaServer.initZk(KafkaServer.scala:326)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:187)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
    at kafka.Kafka$.main(Kafka.scala:67)
    at kafka.Kafka.main(Kafka.scala)
[2018-02-16 12:32:49,030] INFO shutting down (kafka.server.KafkaServer)

[solsson]

Maybe this is a generic zookeeper/kafka troubleshooting case? The zookeeper pod name is different from this repo. Is there a fork with your changes?

[csarora]

No , i do not have a fork, i kept zookeeper as name instead of zoo and both kafka zookeeper deployed in default namespace. can you give me any direction on how can i debug what can i check? as if i have 1 zookeeper kafka able to connect but when multiple it fails,

[solsson]

All I can say then is that I haven't seen an issue like that before. Naming changes may sound cosmetic, but there's a lot that depend on naming.

[Russell-IO]

@csarora check how many client connections ZK is allowing, try setting this to 0 for troubleshooting