search

cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.
https://www.cockroachlabs.com
Other
29.51k stars 3.7k forks source link

storage: alter default cluster setting for sstable compression algorithm #123953

Open nicktrav opened 1 month ago

nicktrav commented 1 month ago

Set storage.sstable.compression_algorithm to zstd.

There are likely some tests that will need to be altered to take into account SSTable size after being compressed with zstd, rather than snappy.

Jira issue: CRDB-38625

nicktrav commented 3 weeks ago

I'm noticing a material increase in CPU time when using zstd as the default (#125463). This was causing tests to timeout.

The following shows a CPU profile with and without the default change, focused in on compressBlock.

darwin-aarch64 Snappy: ``` $ ./dev test ./pkg/kv/kvclient/kvstreamer --filter TestStreamerVaryingResponseSizes --ignore-cache --test-args='-test.cpuprofile=cpu.out.before' ``` CPU time 30ms. Screenshot 2024-06-10 at 7 38 10 PM ZSTD: ``` $ ./dev test ./pkg/kv/kvclient/kvstreamer --filter TestStreamerVaryingResponseSizes --ignore-cache --test-args='-test.cpuprofile=cpu.out.after' ``` CPU time 610ms. Screenshot 2024-06-10 at 7 38 00 PM
linux-x86_64 Snappy: CPU time 810ms. Screenshot 2024-06-10 at 8 06 09 PM ZSTD: CPU time 9.54s. Screenshot 2024-06-10 at 8 06 17 PM

@jbowens - I recall you saying that you had some tuning params up your sleeve that you thought might be worth looking into?

I put the profiles here (internal).