Eventual-Inc / Daft

Distributed DataFrame for Python designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
2.05k stars 139 forks source link

[PERF] [Catalogs] Push partition filter expressions into catalog file URI fetches #1953

Open clarkzinzow opened 6 months ago

clarkzinzow commented 6 months ago

Catalog SDKs expose basic filter expression APIs when fetching the file URIs for reading a particular table, where filters on partition columns can be used to prune the file URIs returned. This can end up eliding a lot of work by the catalog implementations when reading and parsing their metadata files.

samster25 commented 6 months ago

blocked by https://github.com/Eventual-Inc/Daft/issues/1800