search

getindata / kedro-airflow-k8s

Kedro Plugin to support running pipelines on Kubernetes using Airflow.
https://kedro-airflow-k8s.readthedocs.io
Apache License 2.0
29 stars 11 forks source link

Add run-once command to the plugin #13

Closed em-pe closed 3 years ago

em-pe commented 3 years ago

The command should upload the DAG to specified DAG folder and use Airflow API to trigger it immediately. There may be a need to implement additional logic to wait for the dag to be loaded.

Support for config flag wait_for_completion is optional.

Check kedro kubeflow plugin for reference.

em-pe commented 3 years ago

When writing the code for deployment of DAGs to remote location consider using fsspec so it can be used with different filesystems (gcs, s3, local)

michalzelechowski-getindata commented 3 years ago

Support for config flag wait_for_completion is optional.

How about separate issue for this flag support?

michalzelechowski-getindata commented 3 years ago

Related issue #26 created.