search

lni / dragonboat

A feature complete and high performance multi-group Raft library in Go.
Apache License 2.0
5.06k stars 541 forks source link

Read after leader elected return: request dropped as the cluster is not ready #183

Closed lebovski closed 3 years ago

lebovski commented 3 years ago

If I try to SyncRead after leader elected first time, I get error (request dropped as the cluster is not ready). What is the correct way to wait for the cluster to be ready?

Dragonboat version

v3.2.8

Expected behavior

SyncRead successful.

Actual behavior

Error: request dropped as the cluster is not ready

Steps to reproduce the behavior

  1. Create listener, and add SyncRead action on leader elected.
  2. Start nodes.
lni commented 3 years ago

This is because a ReadIndex backed read can only be requested after the new leader figures out what is the current commit value. The new leader achieve that by proposing a noop entry immediately after becoming the leader.

I'd recommend to always add code logic to handle such "cluster is not ready" and timeout failures, as there is no guarantee at any moment that read/proposal can be successfully completed, you always need to be prepared for such failures. As an example, assuming there is another event listener provided by dragonboat for indicating that the commit value is now available to the new leader and thus ReadIndex can be requested, if your program hook up with that event and make ReadIndex read request, you can still end up having the same "cluster is not ready" error in some extreme cases, one possible situation is that the new leader could lose the quorum due to some network partition issues and be forced to start a new election and get re-elected again (as the network partition gets resolved shortly before/during the new election).

That being said, I'd happy to accept PR for adding a new event listener which notifies the application that the cluster is ready for accepting ReadIndex requests. @lebovski would you like to provide a PR for that? Thanks.

lebovski commented 3 years ago

Thanks a lot for the recommendations. Now I don't have much time, but if I have the opportunity, I will try to prepare a PR.