Open humutkazan opened 9 months ago
ZK is getting online then offline and then finally online. After the last successful connection, no progress was observed.
I suspect that the initial request to ZK is somehow hung, and since no timeout is set on ZK client, it waits indefinitely. Guessing this based on this logline:
2023-12-01T05:15:53,975 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
Can that be the reason?
What version of zookeeper are you using? And did you check your zookeeper quorum is actually configured correctly and stable?
It is 3.8.1 ("zookeeper.version" : "3.8.1-74db005175a4ec545697012f9069cb9dcc8cdda7, built on 2023-01-25 16:31 UTC") In this case we're running a single instance of zookeeper.
Problem is reproducable by just restarting zookeeper. Even coordinator detects that zookeeper is back, it is not elected leader again.
I observe the same when with multiple coordinator instances as well.
Try downgrading to 3.6.x
Try downgrading to 3.6.x
I tried with 3.6.0 but nothing seemed to change.
Seeing the same issue with druid 27 + zookeeper 3.7.2 and druid 29.01 + zookeeper 3.8.4
We have encountered an issue recently and we think that it was related with leadership election not happening. You can find the coordinator log file attached.
In the logs you can see the log that says
2023-12-01T05:06:05,659 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow! Starting coordination in [PT10S].
but after some time during the day it says
15:23,973 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am no longer the leader...
and from this point on you could see the logs like
2023-12-01T05:15:57,065 ERROR [KafkaSupervisor-enriched-analog-Reporting-0] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Failed to get task runner because I'm not the leader!
connection time out exceptions etc.
DRUID Version = 27.0.0
0112.log