search

ihedvall / mdflib

Implementation of the ASAM MDF data file.
https://ihedvall.github.io/mdflib/
MIT License
75 stars 31 forks source link

Add support for MDF 4.2 column storage #64

Closed ihedvall closed 11 months ago

ihedvall commented 11 months ago

Adding support for column storage according to new MDF 4.2 standard. It's actually the write support that is prioritized as it can be an Apache Arrow plugin/extension

ihedvall commented 11 months ago

After reading the specification, the column-oriented storage of 4.2, is just splitting up the record into a data and a invalidation block. There is no meaning to support this for writing as it doesn't speed up things very much. It's better to support sorting according to 4.0..4.1 version.

Fixed a solution for RV and RI that should work on DV and DI blocks as well. But the problem is that no reference files exists, so it is currently no way to validate the code.

I propose we wait with the read support of column-oriented storage until we have an example file.