search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
19 stars 3 forks source link

Support Snowflake backend #184

Closed windiana42 closed 5 months ago

windiana42 commented 5 months ago

Add support for Snowflake database as backend.

windiana42 commented 5 months ago

As far as I see, the currently newest snowflake-sqlalchemy 1.5.3 version has two problems: 1) It only supports SQLAlchemy < 2 2) Just adding it to our pyproject.toml results in poetry lock not terminating. We have a general problem with poetry not supporting optional dependencies where a resolution for all optional dependencies together does not exist. But even globally setting SQLAlchemy < 2 dependency, the resolution does not terminate.