etcd-io / raft

Raft library for maintaining a replicated state machine
Apache License 2.0
666 stars 164 forks source link

Remove node #94

Closed kotyara85 closed 9 months ago

kotyara85 commented 1 year ago

Hi there, Is it even possible to remove node completely with this package?

I use a 3rd party wrapper around this raft package which sends this -

  c := raftpb.ConfChange{
        Type:    raftpb.ConfChangeRemoveNode,
        NodeID:  uint64(*node.NodeId),
      }

When I delete a node and then it's data directory, it's still present on the raft members list, raft leader doesn't trying to replicate data to it so I'm assuming the node was removed.

When I'm trying to join the same node to the cluster again it returns - node already joined under another ID

Thanks

pav-kv commented 1 year ago

@kotyara85 It's hard to read your request, could you add some details? How is the 3rd party wrapper structured, and what does it try to achieve?

When I delete a node and then it's data directory, it's still present on the raft members list

Where is this list? Is this raft's list (then please point at the structure/field definition containing this list)? Or is this the 3rd party's list?

As I understand, after a node is removed from raft, it's the application's responsibility to then remove the corresponding data, it's not reusable. If you want a node to rejoin, it has to have a different node ID (even if it's physically the same node). Otherwise, some replicas (who might have still not seen the old node ID removal) can be confused by seeing a node that reuses the ID but behaves differently compared to what would be expected from the old node.

kotyara85 commented 1 year ago

hi @pavelkalinnikov I'm using this package - https://github.com/shaj13/raft/blob/main/node.go#L366-L399

  1. I call this method to removemember and it's being marked as removed in raft config
  2. Removing data directory and trying reconnect node with the same address and ID
  3. This error occurs - rpc error: code = Unknown desc = raft: member 2 removed, raft: node state not found

Thanks

pav-kv commented 9 months ago

This question should be addressed to the third-party package that you are using. https://github.com/shaj13/raft