kv: remove follow-the-workload lease rebalancing?

[nvanbenschoten]

Follow-the-workload has seen its time in the sun. Early in CRDB's history, it was part of the recommended topology patterns for multi-region clusters. However, it was never very good at its job. This is particularly true because follow-the-workload relies on schemas happening to have been organized in a way that groups rows with the same locality affinity together in the KV keyspace so that they end up on the same range. This requirement for the feature to be effective was never documented and so it's doubtful that users ever achieved much success using it to automatically detect and react to sustained locality affinity. At best then, it was useful to automatically react to diurnal load movement which was otherwise locality agnostic.

With lease preferences, a new collection of SQL-level MR abstractions to precisely control data locality, REGIONAL BY ROW auto-rehoming, and an ever-expanding focus on performance predictability, it's not clear that it still has a role in the system. It's also not clear that any customers still use the feature. And yet, there is a performance and maintenance cost to retaining the feature.

Is it time to delete it?

Jira issue: CRDB-23905

[kvoli]

If this is removed, we would also be able to remove the locality tracking which is expensive.

[a-robinson]

In practice, one case that I've found to still be better handled by follow-the-workload than by fixed lease preferences has been when one particular region in a multi-region cluster is overloaded. I had a cluster where one region would get so overloaded that latencies would degrade well beyond an acceptable threshold, but leases for the hot ranges stayed in that region because of a lease preference.

Removing the lease preference meant that in theory latency would be less predictable due to the extra network hop, but in practice latencies dropped way back down because the load-based lease rebalancer was able to move some of the leases to less hot regions nearby. Needing to occasionally wait on the tens of millisecond RTT between regions was much better than regularly having latencies in the high hundreds of milliseconds in the overloaded region.

---

Of course, that being said, that's arguably more a bug in the current implementation of lease preferences than a strong reason to keep follow-the-workload around.

[a-robinson]

Also, for context, that happened on a v22.1 cluster. I don't know if any of the behavior described would have been fixed by any changes since that release.