search

getindata / kedro-airflow-k8s

Kedro Plugin to support running pipelines on Kubernetes using Airflow.
https://kedro-airflow-k8s.readthedocs.io
Apache License 2.0
29 stars 11 forks source link

Replace hardcoded '/opt' in project_path in spark job with PROJECT_HOME env #128

Closed lmaczulajtys closed 2 years ago

lmaczulajtys commented 2 years ago

In Spark jobs, init script already sets PROJECT_HOME variable.

Line: https://github.com/getindata/kedro-airflow-k8s/blob/83e7473c9825724b17571cd13b7ad53925e6e9f6/kedro_airflow_k8s/airflow_spark_task_template.j2#L9 should be replaced with something like this: project_path = os.getenv('PROJECT_HOME','/opt/{{ project_name }}')

In some cases, I need to use different path for project. Thanks to that change, I could use spark.yarn.appMasterEnv.PROJECT_HOME in operator factory to change default /opt path.

em-pe commented 2 years ago

@lmaczulajtys would you mind submitting a PR for that one? Something like project_path = os.getenv('PROJECT_HOME','/opt/{{ project_name }}') I'd approve right away.

Thanks in advance!