An example use case - Multiple raft cluster with in the server cluster

eBay / NuRaft

C++ implementation of Raft core logic as a replication library

Apache License 2.0

997 stars 236 forks source link

An example use case - Multiple raft cluster with in the server cluster #64

Open yashhema opened 4 years ago

yashhema commented 4 years ago

I asked the same question in cornerstone github. Just wanted to get your inputs, how can we implement a scenario Can you please provide your inputs - Consider I have 5 servers (S1,S2,S3,S4,S5) , and 7 tasks which are running on these servers. Each task needs 2 replicas. For task T1 - I can have S1,S2,S3 where S1 is the leader For task T2 - I can have S2,S1,S3 where S2 is the leader and so on. Its possible one of the server goes down, then I should be able to add one of the existing server like say if S1 goes down, I should be able to use S4 instead. Any guidelines or pointers will be really appreciated

greensky00 commented 4 years ago

Hi @yashhema

Can you please elaborate more? Here is my understanding, please correct me If I misunderstood:

There are 5 servers, which will be shared for running 7 tasks.
Each task will pick 3 servers (out of 5) and organize a Raft group. Each group is independent each other, and will do quorum write (commit after being agreed by at least 2 replicas in the group).
A server can be members of different groups at a time.

If so, once one node goes down, you can contact the current leader for each group, remove (by calling remove_srv) the node currently offline, and then add a new node (by calling add_srv). The new node will automatically sync up with leader using snapshot or log. In the meantime (while you remove previous node and new node is catching up), existing groups will be still available as long as remaining 2 nodes are alive.

yashhema commented 4 years ago

Hello, Yes , your understanding is correct. Main idea is that I should be running one exe per server and with in the same exe - I can manage multiple raft groups (using the same port , something like https://github.com/atomix/atomix) Any suggestion are welcome

greensky00 commented 4 years ago

If that's the case, you can remove the problematic node and then add the other one. There will be automatic online sync-up once you add a node to existing running cluster. And you need to make sure that each Raft cluster (for each task) should have separate Raft log. Thanks.

ZTJiu commented 4 years ago

Hi @greensky00 , thanks for your great work!

Does each Raft cluster need a separate nuraft::raft_server instance? If so, could they use the same port for each Raft cluster?

greensky00 commented 4 years ago

@ZTJiu Yes, you need separate raft_server instances. And if they are running in the same process, they should use different listening ports.