search

BasPH / pylint-airflow

Pylint plugin for static code analysis on Airflow code
MIT License
90 stars 13 forks source link

Question with C8306 For consistency match the DAG filename with the dag_id #2

Open louisguitton opened 5 years ago

louisguitton commented 5 years ago

I personally version my DAGs (when we add tasks or change start_date) using only the dag_id, but leaving the filename.

For example

# dags/mailed_report.py

mailed_report = DAG(
    dag_id='mailed_report_v2',
    default_args=default_args,
    catchup=True,
    template_searchpath=['/usr/local/airflow/sql'],
    schedule_interval='@daily'
)

This triggers C8306. I'm curious what your thoughts are: should the filename be versioned as well? or could the match in https://github.com/BasPH/pylint-airflow/blob/master/src/pylint_airflow/checkers/dag.py#L121 be more flexible?

Here is a quick and dirty way to show you what I mean

dagid_without_version =  f"{dagid}".split("_v")[0]
expected_filename = f"{dagid_without_version}.py"
BasPH commented 5 years ago

Thanks for pointing this out. I have seen other people version their DAGs too, it would indeed make sense to allow the DAG to start with the filename and have a postfix. Perhaps by checking if the dag_id is contained in the filename or vice versa, to allow for versions/other postfixes in either.