Roll-up Leaves Command

[osopardo1]

Small files problem is a hot and complex topic in Data Lakehouse systems nowadays. High writing workloads can deteriorate the Data Layout by producing lots of small files, and increasing significantly the cost of listing and reading from the object storage.

One solution is Compaction, which uses a bin-packing algorithm to group small files and produce bigger ones. But this is a way too generic methodology for Qbeast, since each file belongs to a cube, and we need to add another grouping layer on top.

Another solution that we can think of is what we call roll-up.
The roll-up recursively group and send sibling payloads to their parent cubes as long as the resulting size is under a certain threshold, resulting in fewer and larger blocks to write.

Here's a visual example of rolling up leaves cubes of the tree.

We implemented it in the Vanilla version of the code (#146 ) as well as in the Improved Double Pass (#147), and, although the solution on the Improved Double Pass showed better reading results of TPC-DS workload, there's a penalty to consider at writing time. A bigger portion of computation and resources are used to analyze the tree structure and modify the final index results.

Instead of merging this into the base implementation, we are considering making an external command that compacts those leaves into their parents and outputs a better file structure.

It could be something like:

QbeastTable.rollUp(tableID, maxFileSize...)

[osopardo1]

Does this implementation still makes sense due to the current #173 approach? @Jiaweihu08

[Jiaweihu08]

Does this implementation still makes sense due to the current #173 approach? @Jiaweihu08

I think it does, staging enforces a lower-bound for the writes but still, the problem of leaves having smaller sizes remains the same.

[osopardo1]

Okei, I am going to keep it alive for discussion/documentation purposes.

[osopardo1]

Since new developments on #210 would include a Rollup code in the main branch, I peacefully closed this issue.