Add zipped compression for fixed-size-list with nullable items

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

https://lancedb.github.io/lance/

Apache License 2.0

3.97k stars 224 forks source link

Add zipped compression for fixed-size-list with nullable items #3111

Open westonpace opened 1 week ago

westonpace commented 1 week ago

The validity bitmap needs to be zipped in with the values. This process is described (but not implemented) in FslPerValueCompressor