A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
with Flow() as f:
with Stage("stage_1") as s1:
...
with Stage("stage_2") as s2:
task_y = ...
....
# Run entire flow
f.run()
# Run only stage 1
f.run(s1)
# Run only task y
f.run(task_y)
Example code:
Checklist
CHANGELOG.rstentry