workload: dynamic hotkey workload

[kvoli]

In cases where there is a frequently accessed key span that shifts along the keyspace, cockroachdb often develops hot spots on stores that contain the frontier of this "moving hotkey".

We currently do not have any workloads that are representative of this type of workload, despite it being the source of repeated issues in the wild.

It would be desirable to have a workload to benchmark cockroach against that exhibits these pathological characteristics.

The solution is to add a moving hotkey workload that progresses sequentially up the keyspace.

[x] = current key access

t=0 0..[x]..1000.......2000.......3000.....
t=1 0.......1000..[x]..2000.......3000.....
t=2 0.......1000.......2000..[x]..3000.....
t=3 0.......1000.......2000..[x]..3000...[x]

Specifically, writes should be point writes that progress uniquely up the keyspace with each write. Reads should be scans, that scan back the last X writes.

The configurable variables should be:

scan length:    num keys to scan back from current frontier
write speed:    the rate at which to progress the write frontier
scan frequency: the rate at which to scan backwards

If these may be mapped to existing KV workload variables that would be ideal, such as cycle length, concurrency and read ratio.

An a alternative is a more general dyanmic hotkey workload, where the hotspot is determined according to a distribution rather than sequentially moving. Another alternative is to adjust the existing sequential key access generator in KV to allow for reads bounded behind the frontier.

Jira issue: CRDB-16235

[shralex]

@kvoli do you intend to merge this ?

[kvoli]

@kvoli do you intend to merge this ?

I do intend to merge this. I don't intend to merge within the next 2-3 weeks. I believe I'll have some more time to clean this up after clearing GA blockers then :)