search

lni / dragonboat

A feature complete and high performance multi-group Raft library in Go.
Apache License 2.0
4.98k stars 533 forks source link

[Question] Adding removed node back to cluster #350

Closed antmat closed 1 month ago

antmat commented 3 months ago

Hi! We are trying to use dragonboat and we have usecase when node is being removed from the cluster, cleared and after some time plugged back to the cluster. Dragonboat states that "Requesting a removed node back to the Raft cluster will always be rejected.", however we wanted to map our own unique entity one to one to nodeID. In this case it would still be the same and nodeID addition will be rejected. Can you please explain what is the reason for this limitation in case we start a fresh node with the same ID? Thanks in advance

kevburnsjr commented 1 month ago

Seems to me like disallowing the reuse of node IDs just simplifies the protocol and implementation by not having to keep track of extra metadata. If you did want to support node ID reuse, you would have to keep track of a sort of node generation ID in order to allow a node to correctly transition from a closed state to an open state.

This maybe makes more sense in v4 terminology where nodes are called replicas. If you remove a replica from the shard and try to add it back, you have to pull a fresh snapshot and are thus creating a new replica which is already a standard procedure. Best to just march forward with your replica IDs and not try to assign them any special meaning.

lni commented 1 month ago

hi @antmat, as @kevburnsjr explained, allowing removed replicas to be added back with the exact same ID would make the protocol and implementation more complicated, there would be a lot more corner cases that must be carefully addressed.

I can be wrong as it has been a long time, but from memory, etcd requires the ID not to be reused as well.