Scanner tech debt - Githubissues

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

https://lancedb.github.io/lance/

Apache License 2.0

3.95k stars 220 forks source link

Scanner tech debt #1992

Closed chebbyChefNEQ closed 8 months ago

chebbyChefNEQ commented 8 months ago

Problems:

[ ] schema handling and planning are done separately -- we should just make the plan the source of truth of schema
[ ] metadata is returned sometimes but not all times. Make this behavior more consistent.
[ ] make _rowid and _distance less awkward to deal with -- so that select([]).with_row_id() is just select ("_rowid") We should reserve that column name.
[ ] don't return _distance unless select * or select _distance

eddyxu commented 8 months ago

I think we should also consider what logic should be in lance and what should be in lancedb (as vectordb).

Lance itself as format could be more generic.

wjones127 commented 8 months ago

We already have an issue here: https://github.com/lancedb/lance/issues/1782

I've folded these items into that issue.