search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Reorganize dependencies #80

Closed windiana42 closed 1 year ago

windiana42 commented 1 year ago

-[ ] Remove extra dependencies which are only needed for integration tests. -[ ] Create separate test suite for integration tests outside tests/ folder -[ ] Trigger integration tests in CI (with varying dependencies/requirements.txt files)

windiana42 commented 1 year ago

We were considering to move from poetry to conda/mamba/micromamba/flit altogether, but deprioritized this step. We were considering to issue conda build and conda based integration tests in CI without changing from poetry to conda, but concluded that we want to see an issue on client project first where conda vs. pip installation breaks and not waste build cycles / setup time until then.

NMAC427 commented 1 year ago

Closed by #81