TileDB-Inc / TileDB

The Universal Storage Engine
https://tiledb.com
MIT License
1.87k stars 185 forks source link

Maximum fragment size consolidation parameter #2296

Open gatesn opened 3 years ago

gatesn commented 3 years ago

There's an example in the docs of how consolidation can somewhat approximate LSM trees.

It would be helpful when operating in such a mode (consolidating after every write) to define a maximum fragment size preventing overly expensive consolidation. More expensive / unbounded consolidation can then be run manually at another time.

gatesn commented 3 years ago

Also having a dry-run mode would be really useful to experiment with consolidation parameters

stavrospapadopoulos commented 3 years ago

Thanks for raising this, it is already in our roadmap.