feat: compact after merge sort

databendlabs / databend

𝗗𝗮𝘁𝗮, 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 & 𝗔𝗜. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com

https://docs.databend.com

Other

7.85k stars 750 forks source link

Closed zhyass closed 2 months ago

zhyass commented 2 months ago

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Run compact after merge sort during recluster

[ ] Bug Fix (non-breaking change which fixes an issue)
[x] New Feature (non-breaking change which adds functionality)
[ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)
[ ] Documentation Update
[ ] Refactoring
[ ] Performance Improvement
[ ] Other (please describe):

This change is

what-the-diff[bot] commented 2 months ago

Addition of a new compact block process A new file titled transform_block_compact_without_split.rs was introduced. This houses the newly structured BlockCompactorWithoutSplit.
Implementation of the Compactor trait In order to cater to block compacting needs, this trait has been implemented in the newly added file.
Enhancement to the builder reclustering process Some adjustments have been made to the builder_recluster.rs file, which now includes BlockCompactorWithoutSplit and TransformCompact. As a result, the pipeline will compact data blocks post-merge sort.
Test file modifications Some changes have been applied to 09_0008_fuse_optimize_table.test file , more specifically removing the insert into t15 values(5) statement and then adding a fresh insert into t15 values(-7) statement.

github-actions[bot] commented 2 months ago

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] commented 2 months ago

github-actions[bot] commented 2 months ago

note: this image tag is only available for internal use, please check the internal doc for more details.

github-actions[bot] commented 2 months ago