astronomer / dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Apache License 2.0
1.21k stars 183 forks source link

[bug] Timetables issue #287

Open tatiana opened 1 week ago

tatiana commented 1 week ago

This feedback was reported by Karen Braganza on DAG Factory 0.19.0:

Timetables that do not expect a params argument will result in the following import error.

Broken DAG: [/usr/local/airflow/dags/df.py]
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/dagfactory/dagbuilder.py", line 313, in make_timetable
    schedule: Timetable = timetable_obj(**timetable_params)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: plugins.timetables.workday.UnevenIntervalsTimetable() argument after ** must be a mapping, not NoneType

In traditional Airflow, you can create a custom timetable with or without params like below.

schedule=UnevenIntervalsTimetable()
OR
schedule=CronTriggerTimetable("0 18 * * 5", timezone="UTC")

In DAG factory, only the second type of timetable will work.

timetable:
    callable: plugins.timetables.workday.UnevenIntervalsTimetable

OR

timetable:
    callable: airflow.timetables.trigger.CronTriggerTimetable
    params: 
        cron: "0 18 * * 5"
        timetable: "UTC"

The first timetable will produce the TypeError I mentioned because if the params dictionary is not available, the make_timetable() function in dagbuilder.py will receive a None value as the argument for its timetable_params parameter (source code).

dag_kwargs["timetable"] = DagBuilder.make_timetable(
       timetable_args.get("callable"), timetable_args.get("params"))

Since it is expecting a dictionary, timetable_params is passed as timetable_params to the timetable object within make_timetable(). If None is passed as timetable_params instead of a dictionary, a TypeError is produced (source code).

schedule: Timetable = timetable_obj(**timetable_params)

My current workaround is to create a dummy argument in my custom timetable to pass in the params dictionary. Users should be able to use custom timetables without params or a workaround.