oconnor663 / bao

an implementation of BLAKE3 verified streaming
Other
479 stars 23 forks source link

consider an encoded format with empty space for 64 parent nodes at the front #16

Closed oconnor663 closed 6 years ago

oconnor663 commented 6 years ago

That would eliminate the need for the flipping step at the end of (non-memory-mapped) encoding. It could allow efficient appends to an encoded file.

Downsides:

Best alternative: Use an outboard tree, which you incrementally update (not currently implemented). When you need extra space at the front of the tree, just rewrite the whole thing. The overhead will be small relative to all the appending that's happening to the raw output file.

Correction: It's not enough to just have space for parents at the front. You also need space for parents in front of every partial tree along the right edge. (Not 64, but some smaller number corresponding to the maximum size of that subtree.) So this sort of design is actually fairly involved, and not just a tweak to header parsing.