feat: add bitpack encoding for LanceV2

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..

https://lancedb.github.io/lance/

Apache License 2.0

3.47k stars 180 forks source link

feat: add bitpack encoding for LanceV2 #2333

Open albertlockett opened 1 month ago

albertlockett commented 1 month ago

Work in progress

TODO

[ ] improve tests
[ ] support signed types
[ ] handle case where buffer is all 0s
[ ] handle case where num compressed bits = num uncompressed bits

github-actions[bot] commented 1 month ago

ACTION NEEDED

Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.