Eventual-Inc / Daft

Distributed data engine for Python/SQL designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
2.34k stars 164 forks source link

[PERF] [Catalogs] Push partition filter expressions into catalog file URI fetches #1953

Open clarkzinzow opened 8 months ago

clarkzinzow commented 8 months ago

Catalog SDKs expose basic filter expression APIs when fetching the file URIs for reading a particular table, where filters on partition columns can be used to prune the file URIs returned. This can end up eliding a lot of work by the catalog implementations when reading and parsing their metadata files.

samster25 commented 8 months ago

blocked by https://github.com/Eventual-Inc/Daft/issues/1800