search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
12 stars 2 forks source link

Emergency Release 0.7.2: Implemented config for max_copy_operations, pool_size, pool_timeout. #168

Closed windiana42 closed 3 months ago

windiana42 commented 3 months ago

SQLTableStore, now supports parameters max_concurrent_copy_operations, sqlalchemy_pool_size, sqlalchemy_pool_timeout.

We encountered the situation that the pipeline aborted because a sa.inspect() call timed out because the full sqlalchemy connection pool was occupied copying tables between schemas.

Checklist