1xyz / coolbeans

Coolbeans is a distributed work queue that implements the beanstalkd protocol.
Mozilla Public License 2.0
67 stars 6 forks source link

start a cluster with snapshotted data fails #22

Closed 1xyz closed 4 years ago

1xyz commented 4 years ago

Problem: when an assigned bootstrap node attempts to bootstrap an existing cluster, it fails with bootstrap only works on new clusters

Repro: steps 1) I have a three node cluster, but none of the nodes are running. 2) Ensure that the three node cluster has snapshot data from a previous run. For example: a previous run of make run-cluster generated data in /tmp/bean* 3) Run make run-cluster. The bootstrap node fails to bootstrap the cluster with bootstrap only works on new clusters

Expected: the bootstrap to start the cluster.

Root cause: Since raft performs bootstrap for new clusters only. Fix detail: Only the bootstrap node can fails with raft.ErrCantBootstrap. In that case of the bootstrap document, we are going to ignore this error and continue to to start the Grpc server allowing other peers to connect


// called on an un-bootstrapped Raft instance after it has been created. This
// should only be called at the beginning of time for the cluster with an
// identical configuration listing all Voter servers. There is no need to
// bootstrap Nonvoter and Staging servers.
//
// A cluster can only be bootstrapped once from a single participating Voter
// server. Any further attempts to bootstrap will return an error that can be
// safely ignored.
//
// One sane approach is to bootstrap a single server with a configuration
// listing just itself as a Voter, then invoke AddVoter() on it to add other
// servers to the cluster.
func (r *Raft) BootstrapCluster(configuration Configuration) Future {```