search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Support more table stores (i.e. for parquet files or apache arrow dataframes in-memory) #208

Open adzcai opened 2 months ago

adzcai commented 2 months ago

Currently the backend supports either a database backend or a blob backend. It would be nice to enable materializing other types of files, notably Parquet files, as well as blobs. This might enable pydiverse.pipedag to also be used in more lightweight pipelines that don't require the overhead of a SQL backend and can work directly with disk files instead.

windiana42 commented 1 month ago

In deed it is planned to create table backends that either store tables in parquet files on disk or blob store (i.e. S3) or in memory as apache arrow dataframes. => I will rename the issue to reflect the aspect that this request mostly touches extention of more table stores.