alfred-landrum
commented
4 years ago The zdx file would become a concatenation of streams:
- a initial stream, the so-called "super-block", implemented as a zng record with a fixed size, that has a version identifier, and an offset to the btree table-of-contents stream.
- the bulk index key stream, that contains all keys
- btree table-of-contents: contains offsets to each separate btree level stream
- btree level streams
Creating the zdx would work like this:
- first write the super-block zng record, recording the btree offset as zero
- then append the bulk index stream; separate temporary files are used to record the btree levels
- create the btree table-of-contents, and append it,
- finally append the the btree levels
To use this to find keys, a zdx reader would read the zdx file, read the super-block to follow the offsets to the btree toc, then execute a btree search.
philrz
Filing from a live discussion from https://github.com/brimsec/zq/pull/600 : We could implement the zdx bundle as a single file, with potentially very little work, by concatenating the btree files with base file. In this case, the offsets stored would be interpreted as offsets in the next stream of the overall zdx file.