search

astronomer / dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Apache License 2.0
1.21k stars 183 forks source link

[Bug] load_yaml_dags does not call clean_dags #291

Open wearpants opened 1 week ago

wearpants commented 1 week ago

DAG Factory version

0.19.0, 0.20.0

airflow version

2.9

Python version

3.11

Deployment

Other Docker-based deployment

Deployment details

No response

What happened?

DAGs loaded from yaml files via load_yaml_dags are not removed from airflow when the yaml file is deleted; this behavior is different than what's recommended in README (clean_dags();generate_dags()`).

Relevant log output

How to reproduce

Create a directory of yaml files and load it, then delete one. It won't disappear from airflow.

Anything else :)?

Not entirely sure I understand what clean_dags is doing, and how to apply it to the multiple-file load_yaml_dags case

Are you willing to submit PR?

Contact Details

No response

cmarteepants commented 1 week ago

I have a question about this one! We also noticed that clean_dags is not invoked, but in practice noticed that when we removed configs for a specific dag, the dag was deleted - likely because the removal is handled by Airflow natively in your version. It's on our list to look into more deeply as we think the function is there for legacy reasons. Did you experience anything different? Meaning, if you removed configs for a specific DAG while using load_yaml_dags, did the dag go away in Airflow?

We're going to create some official docs, and I had planned to switching to the directory method at that time, but if you confirm that the dag does go away, happy to do it now in the readme to avoid confusion.