Default to raft protocol 3, handling the upgrade from protocol 2

[langmartin]

Background

Raft protocol 3 is a requirement for autopilot, so we should default to it in 1.0. We should avoid putting too much effort into managing this protocol upgrade, it's unlikely to be repeated. We should put enough effort into it that upgrading to 1.0 should not be unusually manual or dangerous.

The Bug

1. Follower B is stopped
2. Follower C is stopped
3. B' is restarted on version 1.0
4. It's serf tags are gossiped to the leader A
5. A removes B. The future returns when the configuration change is committed
6. A adds B'
7. A is stopped
8. C' is restarted. C' only knows about servers {A, B} both of which are now gone

The Fix

The bug won't happen if a rolling upgrade waits until the previous membership change has been committed and then next upgrades a machine that has an up-to-date log. For simplicity of tooling, we could just wait until all cluster members have caught up before marking the cluster safe to upgrade.

There are two implementation options: provide operator tooling to allow the cluster administrator to safely perform a manual rolling upgrade, or handle the protocol upgrade internally so that no additional operator burden is imposed.

Operator Tooling

We could bundle a tool with 1.0 that checks the commit index of every machine in the cluster to determine if it's safe to perform the next upgrade. This could just be a query that hits every server in the cluster to ensure that the index changing the configuration has been accepted on all servers in the cluster. In an ordinary operator upgrade where the servers are updated with a rolling upgrade, this check will be true in less time than it takes for stats dashboards to catch up, so a typical operator would just experience a fairly simple but necessary check at every step of the upgrade process.

This check could be generalized to be a new requirement for the upgrade process, where it may be a good place to ship data version/consistency preflight checks in the future.

The main risk of operator tooling is that this 1.0 release, which communicates stability, will require a new manual upgrade step and could cause a cluster to need to be recovered if upgraded too quickly.

Automatic Upgrade

1. upgrade all the servers to 1.0
2. the leader waits until ServersMeetMinimumVersion shows that all servers are running 1.0
3. the leader calls new UpgradeProtocol RPC on a follower
4. the leader's raft library commits to raft a RemoveServer configuration change for that follower
5. that follower receives the RPC and
   1. disconnects
   2. upgrades it's raft instance
   3. rejoins the cluster
6. the leader adds the protocol 3 instance to the raft cluster
7. the leader waits until follower has its log up to date
8. the leader repeats 3-7 for all remaining protocol 2 followers
9. the leader removes itself from the cluster and upgrades locally

More notes:

1. the raft protocol 2 instance must leave the cluster first to avoid the illegal state of two raft servers having the same Addr
2. the raft protocol 3 instance is added to the raft cluster with raft.AddStaging, but that method is a stub that just adds does an AddVoter (there's a todo comment in the raft libarary). Checking the instance's log index is the only way to ensure that it's caught up.
3. waiting for follower to become integrated with the cluster before moving on to the next server is necessary to prevent RemoveServer from decreasing the quorum size. We want to keep the quorum size constant while upgrading.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.