StarRocks / starrocks

StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries.
https://starrocks.io
Apache License 2.0
8.26k stars 1.67k forks source link

[Enhancement] Support zstd level (backport #46976) #47623

Closed mergify[bot] closed 3 days ago

mergify[bot] commented 3 days ago

In this pr, we introduce configable compression level for ZSTD compression type. Valid level is 1-22, default level is 3 (which is same as before)

How to use: CREATE TABLE t ( k BIGINT NOT NULL, v string COMMENT) ENGINE=OLAP DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( \"compression\" = \"zstd(1)\" );

We can define the compression level for zstd using zstd(level) in CREATE TABLE STATEMENT

We load 10000000row from Clickbench dataset, using different level of zstd compression, result as following: level show data
1 103.477 MB
3 (defualt) 100.557 MB
5 98.896 MB
10 96.549 MB
15 95.890 MB
20 91.921 MB
22 91.921 MB

Fixes https://github.com/StarRocks/starrocks/issues/46839

What type of PR is this:

Does this PR entail a change in behavior?

If yes, please specify the type of change:

Checklist:

Bugfix cherry-pick branch check:

How to use: CREATE TABLE t ( k BIGINT NOT NULL, v string COMMENT) ENGINE=OLAP DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( \"compression\" = \"zstd(1)\" );

We can define the compression level for zstd using zstd(level) in CREATE TABLE STATEMENT

We load 10000000row from Clickbench dataset, using different level of zstd compression, result as following: level show data
1 103.477 MB
3 (defualt) 100.557 MB
5 98.896 MB
10 96.549 MB
15 95.890 MB
20 91.921 MB
22 91.921 MB

Fixes https://github.com/StarRocks/starrocks/issues/46839

What type of PR is this:

Does this PR entail a change in behavior?

If yes, please specify the type of change:

Checklist:

sonarcloud[bot] commented 3 days ago

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud