kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.
Apache License 2.0
92 stars 89 forks source link

Extend Kedro-Airflow CLI functionality with optional jinja template #26

Closed fdroessler closed 2 years ago

fdroessler commented 2 years ago

Description

The default Jinja2-Template works well as a base to get started on using kedro with Airflow. However, the fact that a few things are hardcoded in the plugin/template makes it hard to modify a variety of things. I would propose a couple of adjustments in order to allow for more customisability:

I have already implemented those in a branch and was curious if this would be a welcomed PR.

Context

In a recent project I found myself constantly having to adjust the resulting dag files as I was experimenting with a variety of things. I ended up replacing the template file in the site-packages folder at some point which struck me as too hacky. The above changes would allow users to provide their own Jinja2 Templates as well as generating Airflow Dags for multiple pipelines without having to manually rename the output files.

Possible Implementation

Add two more click options (names just examples):

@click.option("-jd", "--jinja-dir", "jinja_template_dir", default=str(Path(__file__).parent))
@click.option("-j", "--jinja-file", "jinja_template", default="airflow_dag_template.j2")

Modify the loader and template respectively:

loader = jinja2.FileSystemLoader(jinja_template_dir)
jinja_env = jinja2.Environment(autoescape=True, loader=loader, lstrip_blocks=True)
jinja_env.filters["slugify"] = slugify
template = jinja_env.get_template(jinja_template)

Adjust the dag_filename if necessary:

if pipeline_name != "__default__":
    dag_filename = f"{package_name}_{pipeline_name}_dag.py"

Looking forward to some feedback on this suggestions.

noklam commented 2 years ago

@fdroessler Thank you for creating this issue, the additional options make sense to me, PR is very welcomed!