search

go-zookeeper / zk

Native ZooKeeper client for Go
BSD 3-Clause "New" or "Revised" License
504 stars 130 forks source link

Watchers are crashing on the zookeeper leader restart #123

Open Hazarath-Reddy opened 6 months ago

Hazarath-Reddy commented 6 months ago

Hi Team,

We have a Zookeeper production cluster with 6 nodes(1 leader,2 followers, 3 observers). We are seeing a drop in watchers if the leader restarts for some reason without drop-in connections. Please see the sequence of actions listed below.

  1. The leader restarted for some reason.
  2. All the followers restarted because of the leader restart to elect new leader.
  3. We see there is no drop in the number of connections. Connections shifted from one ZK VM to another ZK VM.
  4. But we are seeing a drop in the number of watchers created. Each connection that we create is associated with one to twenty watchers depending on the use case.

We would like to understand, is this the the bug, or when ever connection drop happens we need to recreate the watcher?