Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
We should benchmark the latency we get from different block sizes as well as how much concurrency we can get from different object stores.
Because of request rate limits, there's likely to be a tradeoff between latency (better with smaller block sizes) and throughput (better with larger block sizes). Depending on the type of query, we may want to optimize for one or the other. For example, a KNN query will likely want to optimize for latency. Whereas a OLAP scan will want to optimize for throughput. We should be able to provide good automatic defaults based on the query plan. Of course, the user can always input their own settings as needed.
As demonstrated by @westonpace, we can use the tracing module and jeager to instrument the object store and get more insight into how much concurrency we are getting in requests. Blocking this on #1292
wjones127
commented
12 months ago
We should benchmark the latency we get from different block sizes as well as how much concurrency we can get from different object stores.
Because of request rate limits, there's likely to be a tradeoff between latency (better with smaller block sizes) and throughput (better with larger block sizes). Depending on the type of query, we may want to optimize for one or the other. For example, a KNN query will likely want to optimize for latency. Whereas a OLAP scan will want to optimize for throughput. We should be able to provide good automatic defaults based on the query plan. Of course, the user can always input their own settings as needed.