Add zipped compression for fixed-size-list with variable width items

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

https://lancedb.github.io/lance/

Apache License 2.0

3.97k stars 224 forks source link

Add zipped compression for fixed-size-list with variable width items #3112

Open westonpace opened 1 week ago

westonpace commented 1 week ago

See TODO item in FslPerValueCompressor