fix: split if block too big during append

[zhyass]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

1. Refactoring TransformCompact:

   • The TransformCompact was split into two components to improve the modularity and efficiency of the compaction process:
     • BlockCompactBuilder: Responsible for constructing compaction tasks.
     • TransformCompactBlock: Executes the actual compaction in a parallelized manner.

2. Improvement in Compaction Logic:

   • The logic was adjusted to avoid writing excessively large data blocks during compaction.
   • The new structure ensures compaction is done as early as possible during the data writing phase.

3. Block Size Control:

   • The changes aim to fine-tune block sizes during the compaction, ensuring that the resulting blocks are neither too small nor too large, which can impact the efficiency of both storage and read operations.

4. Replace HashMap with BTreeMap in reclustering fetch_max_depth, for stable reclustering effects

5. Compact source data blocks before reclustering, for better performance and clustering

Tests

• [ ] Unit Test
• [x] Logic Test
• [ ] Benchmark Test
• [ ] No Test - Explain why

Type of change

• [x] Bug Fix (non-breaking change which fixes an issue)
• [ ] New Feature (non-breaking change which adds functionality)
• [ ] Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• [ ] Documentation Update
• [ ] Refactoring
• [ ] Performance Improvement
• [ ] Other (please describe):

---

This change is 

[github-actions[bot]]

At least one test kind must be checked in the PR description. @zhyass please update it 🙏.

[github-actions[bot]]

At least one test kind must be checked in the PR description. @zhyass please update it 🙏.

[github-actions[bot]]

Docker Image for PR

• tag: pr-16435-e590446-1725992436

note: this image tag is only available for internal use, please check the internal doc for more details.

[github-actions[bot]]

ClickBench Report

• hits: https://benchmark.databend.com/clickbench/pr/16435/10797840195/hits.html
• internal: https://benchmark.databend.com/clickbench/pr/16435/10797840195/internal.html
• load: https://benchmark.databend.com/clickbench/pr/16435/10797840195/load.html
• tpch: https://benchmark.databend.com/clickbench/pr/16435/10797840195/tpch.html

[github-actions[bot]]

Docker Image for PR

• tag: pr-16435-bc04f62-1726020798

note: this image tag is only available for internal use, please check the internal doc for more details.

[github-actions[bot]]

ClickBench Report

• hits: https://benchmark.databend.com/clickbench/pr/16435/10803448720/hits.html
• internal: https://benchmark.databend.com/clickbench/pr/16435/10803448720/internal.html
• load: https://benchmark.databend.com/clickbench/pr/16435/10803448720/load.html
• tpch: https://benchmark.databend.com/clickbench/pr/16435/10803448720/tpch.html

[github-actions[bot]]

Docker Image for PR

• tag: pr-16435-084e520-1726056170

note: this image tag is only available for internal use, please check the internal doc for more details.

[github-actions[bot]]

ClickBench Report

• hits: https://benchmark.databend.com/clickbench/pr/16435/10810711250/hits.html
• internal: https://benchmark.databend.com/clickbench/pr/16435/10810711250/internal.html
• load: https://benchmark.databend.com/clickbench/pr/16435/10810711250/load.html
• tpch: https://benchmark.databend.com/clickbench/pr/16435/10810711250/tpch.html

[github-actions[bot]]

Docker Image for PR

• tag: pr-16435-7dac011-1726076159

note: this image tag is only available for internal use, please check the internal doc for more details.

[github-actions[bot]]

ClickBench Report

• hits: https://benchmark.databend.com/clickbench/pr/16435/10816309210/hits.html
• internal: https://benchmark.databend.com/clickbench/pr/16435/10816309210/internal.html
• load: https://benchmark.databend.com/clickbench/pr/16435/10816309210/load.html
• tpch: https://benchmark.databend.com/clickbench/pr/16435/10816309210/tpch.html