Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
Problem Statement
Apache Arrow does not support field reference to a
list<struct>
Error:
Expected Behavior
Using
annotations.label
should returns values with typelist<struct<label: str>>
, a subset view of the original annotationslist<struct>