Create a "DAG Authoring Handbook"

[AetherUnbound]

Description

By virtue of being workflows which can be defined in pure Python, there are innumerable ways to set up and author DAGs. Particularly as we get more contributors, it may be useful and prudent to define a set of patterns to use (and to avoid) when authoring DAGs. This document could include information on:

• Where to put the DAG in the codebase
• Naming conventions for the DAG
• Tag conventions
• Variable name standardization (see WordPress/openverse#1507)
• Connection name standardization
• Preferentially using op_kwargs with PythonOperators
• When to use the TaskFlow API (or when it can be used if desired)
• How to write tests for a DAG (e.g. adding it to the list of DAGs to parse)
• Patterns which may exist in the codebase but should be avoided (e.g. get_*_operator functions WordPress/openverse-catalog#238, WordPress/openverse-catalog#301)
• How to choose a start_date for a DAG
• Etc.

It could even potentially include some boilerplate DAG templates (I know that I almost never write a DAG from scratch :sweat_smile:)!

Alternatives

Additional context

Implementation

• [ ] 🙋 I would be interested in implementing this feature.

[zackkrida]

I'd encourage us to think about how much of this could be handled with custom lint rules, templates, and things like that over a manual handbook. Happy to have some documentation about best practices, but the more we can automate the better!

[sarayourfriend]

Some of the syntax and name type requirements could be defined in a flake8 plugin: https://flake8.pycqa.org/en/latest/plugin-development/index.html

Filenames can be handled by https://pypi.org/project/flake8-filename/

A pylint custom checker might be easier to implement than a flake8 plugin though... https://pylint.pycqa.org/en/latest/development_guide/how_tos/custom_checkers.html