ScottLouvau
commented
4 years ago BION is early work. so there's not much documentation in place yet. There was a format description in the wiki, but it looks like it's gone. I'll have to figure out how to restore it. :/ BION isn't a Microsoft Research project.
BION has a few unique design features. It uses illegal UTF-8 bytes - 0xF5 to 0xFF - for all structural tokens. Strings and containers may be length prefixed or use an illegal UTF-8 byte terminator. This allows writing values when the length isn't known in advance, but makes finding the end of things extremely fast. This design also means that string searches (whole values, partial values, and propertyName : value) can be done directly against the file bytes without having to parse and interpret the file, and the structural tokens ensure you can parse backwards to identify where in the document a match was found if you do this.
BION specifies an optional container index, which describes the overall structure of the document and allows readers to select or skip subtrees of the document freely during reading. This can allow readers to choose to partially load large documents and dynamically decide what to defer or skip loading based on the size of the document, all without a full parse.
The current active work is on the "BSOA" offshoot of BION. If you have a consistent schema (a fixed set of object types, with properties which always have the same type), you can use BSOA. BSOA writes values in a columns as blocks, so it's more like formats like Apache Arrow. BSOA provides gigabyte-per-second read and write speeds from a single thread, no data conversion to the in-memory representation, a familiar "with schema" model, like JSON, the ability to add or omit specific columns or tables from an on disk log, and a friendly object model which looks like a normal C# object model but interacts with the columnar data inside, with full mutability on the data inside.
AlgorithmsAreCool
I don't intend to insult anyone's efforts but binary json formats seem like a very well trodden path.
And since the readme doesn't provide much context, it seems confusing.
If this is coming out of Microsoft Research, is there a paper describing it?
If this is an engineering initiative, then how does improve on existing formats such as BSON, CBOR and MSGPACK?