getindata / kedro-airflow-k8s

Kedro Plugin to support running pipelines on Kubernetes using Airflow.
https://kedro-airflow-k8s.readthedocs.io
Apache License 2.0
29 stars 11 forks source link

Select pipeline for DAG generation #54

Closed michalzelechowski-getindata closed 3 years ago

michalzelechowski-getindata commented 3 years ago

Similar to vanilla kedro -p option, which selects pipeline to be operated, airflow-k8s plugin could allow selection of specific pipeline for DAG generation. That functionality could be useful in bigger projects, were pipelines are complex, can be separated but share common nodes.

michalzelechowski-getindata commented 3 years ago

@em-pe Do you think CLI parameter would be sufficient or we should allow this to be part of the config? In case of config, we could use it for separation of runtime parameters.

em-pe commented 3 years ago

@michalzelechowski-getindata CLI should be sufficient I think