search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
13.51k stars 1.29k forks source link

raft topology: (design) external recovery tool for majority loss and group 0 bugs #18791

Open kbr-scylla opened 5 months ago

kbr-scylla commented 5 months ago

To recover from majority loss situation we revert back to gossip mode.

We want to get rid of gossip-based topology operations (https://github.com/scylladb/scylladb/issues/15422) -- for that, we need a different way to perform recovery. I propose we design an external tool which would connect to all remaining live nodes, reconcile their topology metadata (including stuff like tablets), decide which nodes to remove, and coordinate recovery of group 0 state.

This will also help in situations where group0_state_machine got stuck e.g. due to broken invariants due to bugs, like in https://github.com/scylladb/scylladb/issues/20039

cc @gleb-cloudius @tgrabiec @bhalevy @avikivity

kbr-scylla commented 2 months ago

User case where group0_state_machine is halted due to broken invariants (tablets metadata cannot be loaded): https://github.com/scylladb/scylladb/issues/20039

Manual recovery tool would be useful for such case, but simple "copy group 0 state from one node and paste it to others" would not work. It would have to fix the state as well, e.g. by disabling ongoing tablet migrations

kbr-scylla commented 2 months ago

Added "group 0 bugs" to the title