Implement pipeline module (0.5.0)

ericpan64 / pydian

Python framework for developer-friendly data interchange

MIT License

5 stars 0 forks source link

Implement pipeline module (0.5.0) #6

Open ericpan64 opened 1 month ago

ericpan64 commented 1 month ago

Module for running pipelines of data. Differs from validation as focuses on transformation and the various states involved in each step (rather than just checking data). Can add Rule | RuleGroups to stop execution (like assert statements)

Overview

...

High-level design

...

Additional context

Also closes https://github.com/ericpan64/pydian-canvas-fork/issues/41
- Dataframe DAGs are a bit ambitious, though let's see...

ericpan64 commented 1 week ago

Note: Huggingface library uses the words pipeline and pipe already, so keep that in mind (worth looking into how it's used)