search

kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.
Apache License 2.0
84 stars 76 forks source link

kedro-airflow: implement @task and @dag decorators in DAG, generated by plugin #705

Open DimedS opened 1 month ago

DimedS commented 1 month ago

Description

Following the investigation done in #25:

We will implement the @task and @dag decorators from the TaskFlow API into the DAG generated by the kedro-airflow plugin from the Kedro project. This implementation will:

  1. Make the DAG code shorter and cleaner.
  2. Allow us to move away from using KedroOperator, simplifying the user experience and addressing Kedro plugins issue #482.
  3. Demonstrate our commitment to keeping up with current best practices and open the way for potentially more extensive use of TaskFlow in the future.

Possible Implementation

Modify airflow_dag_template.j2 accordingly

noklam commented 1 month ago

This seems contradicts to the conclusion of https://github.com/kedro-org/kedro-plugins/issues/25?

ankatiyar commented 1 month ago

@noklam We discussed a bit more and decided to adopt certain aspects of the TaskFlow Api for now.