search

getindata / kedro-airflow-k8s

Kedro Plugin to support running pipelines on Kubernetes using Airflow.
https://kedro-airflow-k8s.readthedocs.io
Apache License 2.0
29 stars 11 forks source link

Add custom script injection to airflow_spark_task_template #130

Open lmaczulajtys opened 2 years ago

lmaczulajtys commented 2 years ago

It would be great to have functionality similar to

https://github.com/getindata/kedro-airflow-k8s/blob/83e7473c9825724b17571cd13b7ad53925e6e9f6/kedro_airflow_k8s/dataproc_init_script_template.j2#L5

in kedro_airflow_k8s/airflow_spark_task_template.j2. Probably after line with project_path variable.

em-pe commented 2 years ago

@lmaczulajtys there is no active use of this integration on our side currently so if you'd be able to submit the PR that is tested on your premises, we'll take it in, otherwise it may take a while once we pick it up.

You can then reach us over email later on, to arrange a new release.