Catalog using S3 List and Hive partitioning

cloudfuse-io / buzz-rust

Serverless query engine

MIT License

140 stars 11 forks source link

Open rdettai opened 3 years ago

rdettai commented 3 years ago

It would be valuable to make it possible to dynamically create a partitioned catalog from the S3 file structure.

Create a new TableProvider that uses S3 List and the folder structure to create a file table
Implement the SplittableTable that using this provider
Cache this catalog to increase speed on subsequent calls
Add S3 List access to the Fuse

rdettai commented 3 years ago

Buzz now support DeltaLake as source Catalog thanks to #16. This feature might still be relevant to make Buzz more flexible.