add feature of 'private nodes'

mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

MIT License

2.07k stars 100 forks source link

add feature of 'private nodes' #51

Closed ghost closed 3 years ago

ghost commented 4 years ago

Hide 'private' nodes in dependency graph.

For the mara-singer package I added an internal pipeline (https://github.com/hz-lschick/mara-singer/commit/0f5be27cfa511e62b25995b233191885373edbb3) which must be part of the root pipeline in order to use mara_pipelines.ui.cli.run_pipeline.

With this change, the internal _singer pipeline is hidden from the user UI.

ghost commented 3 years ago

Frankly, didn't really like this solution and working now on another one e.g. executing the internal singer pipeline without writing the log output (=event handling) to the database. Such a feature could be used for unit testing as well.