Flexible node selection in `kedro run` command line syntax

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

Apache License 2.0

9.47k stars 875 forks source link

Description

I would like to be able to run nodes across namespaces in a more flexible way.

Context

Let's say I have three identical nodes across three namespaces: red.node, blue.node, yellow.node.

If I want to run them all, my only option is: kedro run --nodes red.node,blue.node,yellow.node

As number of namespaces increase this gets unweildy.

Possible Implementation

Add bash-style or dataset factory style wildcarding to the run command:

kedro run --nodes "*.node"
Or kedro run --nodes "{namespace}.node"

Possible Alternatives

Suggestion from @datajoely:

DBT style syntax for inclusion / exclusion / etc: https://docs.getdbt.com/reference/node-selection/syntax

kedro-org / kedro