Open brucearctor opened 1 year ago
The rules defined in https://github.com/BasPH/pylint-airflow seem to cover all those cases plus a bit more. Would a port of that Pylint plugin be a good start?
I'd had some discussion with @jlaneve on Twitter about this! I'd like to support Airflow-specific rules, but we need guidance on what those rules should contain.
@jlaneve -- any further thoughts here? Is the pylint-airflow plugin a good starting point?
Bas' pylint-airflow is definitely a good start, but worth noting it was designed for Airflow 1.x and we should aim to get ruff supporting Airflow 2.x, so there are likely a few changes we'll want to make. Maybe I can open a draft PR with a few of the easy rules (no duplicate DAG names, no empty DAGs, etc) to get something going, and then we can open issues for specific rules afterwards?
@charliermarsh - what do you think?
@jlaneve - Yeah, that's perfect.
Exactly - I had seen pylint-airflow
, that covers lots of the ideal rules. V2 Airflow is the thing to target, and imagine even ignoring V1 [ for ruff ] is OK.
https://docs.astronomer.io/learn/dag-best-practices https://airflow.apache.org/docs/apache-airflow/stable/faq.html https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html
Some opinionated options (which could certainly be contentious):
with DAG
and @dag
in the same repo owners
DAG as dag
if airflow>X.Yglobals()[dag_id] = dag
if airflow>x.ydag_id
is the same as the filenamestart_date
is a static datetimeSome Astronomer folks might be able to help development :)
Is there a roadmap timeline for this?
what essentially makes the difference between ruff check vs ruff lint lint gives out this error which doesn't seem to get resolved. lint:1:1: E902 No such file or directory (os error 2) any reason?
@usethia your question seems unrelated to this issue. I'll reply anyway but please open a new issue if you need more help to avoid sidetracking this issue.
ruff lint
isn't a Ruff command. Today ruff <path>
is an alias for ruff check <path>
. So the command ruff lint
is the same as ruff check lint
but you don't have a file or directory called lint
in your project. We're about to remove the alias in the next minor version of Ruff because it can be confusing (as in your case).
To lint your project, run ruff check
.
There are numerous problems that could go wrong with an Airflow DAG, ex:
Seems like we might want new rules that could be used to detect these things.