kvserver: simplify replicasByKey handling

[tbg]

Is your feature request related to a problem? Please describe.

In https://github.com/cockroachdb/cockroach/issues/73721 we found a bug in our replica handling that could lead to data corruption. The basic problem is that the Store maintains a mapping of keyspace to Replica/Placeholder but code interacts very directly with it and it's easy to mess it up.

In addition to this, (*Replica).Desc() provides the key bounds of the replica object within the btree, which is deadlock prone (and also means that (*Replica).setDescLocked has an affect on the btree. This is all fairly cryptic and it would likely be better not to require a mutex when looking up a Replica from the tree, and to make any bounds adjustments explicit.

Describe the solution you'd like

Mediate all mutations to the Store's mapping (including handling of uninitialized Replicas) through an API that enforces that Replica insertion or bounds changes go through a ReplicaPlaceholder. In particular,

• shrinking or extending a replica in the btree must happen explicitly via a call through the API (and caller must properly synchronize it with changing r.Desc()).
• split needs to create a placeholder for the RHS atomically with shrinking the left-hand side, then (atomically) upgrade the placeholder to the newly created right-hand side. (Or directly insert the right-hand side; whatever is cleaner).
• merge needs to swap right-hand side for a placeholder that it then absorbs into the left-hand side. (Or directly if possible).

Describe alternatives you've considered

Additional context

Jira issue: CRDB-12057

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!