search

kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.
Apache License 2.0
91 stars 83 forks source link

Deterministic `kedro-airflow` DAGs #380

Closed sbrugman closed 6 months ago

sbrugman commented 11 months ago

Description

The DAGs generated by kedro-airflow are not deterministic, as they are only topologically sorted.

Context

As a user that version controls the DAGs, I would like to have a diff only when the DAG changes (the ordering of the edges does not).

Diff in Edges

Avoiding situations where the diff consists only of edges ordered differently.

Possible Implementation

Sort the edges deterministically in kedro-airflow

Possible Alternatives

(Optional) Describe any alternative solutions or features you've considered.

noklam commented 7 months ago

I can confirm this behavior with kedro-airflow 0.8.0 and kedro 0.19.2.