nicktrav
commented
2 years ago The spikes in increased read throughput correlates strongly with periods during which the auto create stats job is running. Note that we consume more read bandwidth, and as the device is maxed out, we start "stealing" throughput from writes. This is a look at the same time period that you posted:
Zooming out further, we see the same thing, though this time we have much more throughput to consume (we bumped up to 250 MB/s). That said, we still see increased reads stealing some write throughput (note the dips in the green line on the write throughput chart at the bottom when the read throughput increases ):
irfansharif
nvanbenschoten
sumeerbhola
aadityasondhi
https://github.com/cockroachdb/cockroach/pull/81516 adds the
admission/follower-overload/presplit-controlroachtest. In this roachtest, a three node cluster is set up so that two nodes have all leases for a kv0 workload. At the time of writing, kv0 runs with 4mb/s of goodput (400 rate limit * 10k per write). On AWS (where this run took place), on a default EBS volume with throughput limit 125mb/s and 3000 iops (aggregate read+write), this is right at the limit. As a result, n1 and n2 get into mild IO overload territory.It was observed that the nodes with leases consistently read more data from disk (green and orange are n1 and n2)
read mb/s:
Zooming out, we see this pattern:
No splits are occurring at the time. However, the bumps match up well with these bumps in raft log:
The raft log queue processes replicas at a fixed rate throughout these spikes, so it's unclear if it is now simply contending with read activity or if it is itself the cause of read activity.
Overlaying
rate(rocksdb_compacted_bytes_read[$__rate_interval])onto the bytes read shows that compactions are not the driver of the spiky reads on n1 and n2. Quite the opposite, whenever these spikes occur, compactions can't read as quickly as they would like to.Jira issue: CRDB-16492
Epic CRDB-37479