Multi round rebase compaction support

ray-project / deltacat

A portable Pythonic Data Catalog API powered by Ray that brings exabyte-level scalability and fast, ACID-compliant, change-data-capture to your big data workloads.

Apache License 2.0

166 stars 23 forks source link

Multi round rebase compaction support #342

Closed akindu-amazon closed 3 months ago

akindu-amazon commented 3 months ago

Added multi round splitting support:

New num_round parameter in CompactPartitionParams (default is 1)
New helper function that groups uniform_deltas into batches if num_rounds is not 1
New pytests to test aggregation across multiple rounds (drop_duplicates = False) and important asserts (currently not supporting backfill for multiple rounds, hb count should not be 1 for multiple rounds, etc.)
Bug fix in read_delta_file_envelopes