Implement bulk load to snowflake database (the driver has a mechanism to load parquet files via S3)

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.

https://pydiversepipedag.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

12 stars 2 forks source link

Implement bulk load to snowflake database (the driver has a mechanism to load parquet files via S3) #189

Open windiana42 opened 2 months ago

windiana42 commented 2 months ago

Version 0.9.1 introduced snowflake backend, but did not implement bulk upload/download via parquet.