pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.

https://pydiversepipedag.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

15 stars 2 forks source link

Extract cache management functionality out of core.py #32

Closed windiana42 closed 1 year ago

windiana42 commented 1 year ago

This should pave the way to allow implementing n-slot caching without too much change to core.py.

Checklist

[x] Added a CHANGELOG.rst entry

windiana42 commented 1 year ago

@NMAC427 this is just a first try to separate out the caching functionality in a separate class with an info object moved around by the calling code. There is certainly fine-tuning necessary which functionality goes where but it should give an idea about changes I intend.

windiana42 commented 1 year ago

I will merge to main on test green in order to release 0.2.0 as the first version usable in real projects. @NMAC427 sorry, that you didn't find the time to "intervene" here. I am sure we will change this a lot more. But I think it is a step in the right direction.