search

lni / dragonboat

A feature complete and high performance multi-group Raft library in Go.
Apache License 2.0
5.06k stars 541 forks source link

Balancing groups by usage? #238

Closed danthegoodman1 closed 2 years ago

danthegoodman1 commented 2 years ago

Are there any suggestions about how to balance the leaders of (multiple) groups among nodes so that usage (by some metric) is considered when choosing (and changing) leaders?

For example if we are using cockroachdb-style ranges for a kv store, and one range is getting hot, we’d want to split that range. How can we use dragonboat to make sure those 2 new ranges don’t land on the same node?

And for existing groups, how can we move the leader to another node without doing a split?

danthegoodman1 commented 2 years ago

Possibly it seems like RequestLeaderTransfer might be a solution here, although I see #223 might cause issues? (I have not tested yet)

danthegoodman1 commented 2 years ago

Also a combination of removing and adding members to the group, one could coordinate moving of group membership based on custom metrics.

Will need some way to determine who to elect as the next member, and not accidentially add too many members to the group...

danthegoodman1 commented 2 years ago

Seems like maybe something like mangos' SURVEY or even Gossip can be used to determine who is best to add as a member.

lni commented 2 years ago

Are there any suggestions about how to balance the leaders of (multiple) groups among nodes so that usage (by some metric) is considered when choosing (and changing) leaders?

You need to keep monitoring the usage in your code, then request leadership transfer whenever you program believes necessary. Which node to be picked as the new leader should also be determined by your program, probably also based on the collected metrics.

Possibly it seems like RequestLeaderTransfer might be a solution here, although I see https://github.com/lni/dragonboat/issues/223 might cause issues? (I have not tested yet)

was quite busy in the last several weeks, will resolve that soon.

Seems like maybe something like mangos' SURVEY or even Gossip can be used to determine who is best to add as a member.

I think such features are out of the scope of a consensus library. User programs can implement whatever similar features they like, the consensus library itself (e.g. dragonboat here) should just provide the API to allow users to do membership change/leadership transfer.

danthegoodman1 commented 2 years ago

@lni thanks for the feedback. I agree this is definitely out of scope of this package, but wanted to consult the experts on how best to use this package to achieve my goals. It seems with a little time and investigation I was able to find the functionality needed. Thanks for confirming.