kv: scatter doesn't move replicas if there are zone configs

[andrewbaptist]

Scatter is intended to move both leases and replicas, however replicas are not moved. This impacts how index backfills move replicas around and can result in uneven and inefficient systems.

Steps to reproduce:

roachprod create -n12 $CLUSTER
roachprod put $CLUSTER artifacts/cockroach
roachprod start $CLUSTER
roachprod ssh $CLUSTER:1 "./cockroach workload init kv --splits 12 {pgurl:1}"
roachprod ssh $CLUSTER:1 "./cockroach workload run kv  --duration=600s --max-block-bytes=10000 --min-block-bytes=10000 --concurrency=100 {pgurl:1-12}"

After the fill is complete, look at the number of ranges on the kv database, it will likely be ~100. Run a scatter:

ALTER TABLE kv.kv SCATTER;

Pro tip - set the rebalance rate higher:

SET CLUSTER SETTING kv.snapshot_rebalance.max_rate='256MiB';

This will move ~100 replicas. (Side note should this have moved ~300 replicas or only one replica per range)? Add a zone constraint:

ALTER DATABASE kv CONFIGURE ZONE USING constraints = '[+node12]'

Wait for the data to finish moving (check the Data Distribution tab from Advanced Config) Run another scatter and notice that nothing moves and the command completes immediately.

Looking at the distribution log from one of the nodes, notice the following log for all the ranges (shortened for simplicity)

allocatorimpl/allocator_scorer.go:1538 ⋮ *kvpb.AdminScatterRequest  s1: should-rebalance(invalid): locality:‹"cloud=gce,region=us-east1,zone=us-east1-d,node=1"›

Specifically it doesn't scatter because all the replicas attempt to move the replica off of n12 and this fails due to the zone config constraint.

While this is a slightly contrived example, something very similar happens for index backfill if there are any zone configs in place that target certain nodes.

The expected behavior is that the scatter would either move ALL replicas for a range or continue until it finds at least one replica it can move.

Jira issue: CRDB-42285

[blathers-crl[bot]]

Hi @andrewbaptist, please add branch-* labels to identify which branch(es) this C-bug affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[andrewbaptist]

This was due to an invalid zone constraint, closing.