search

astronomer / dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Apache License 2.0
1.21k stars 180 forks source link

Not able to set `on_failure_callback` in default args #177

Open madhavdube opened 1 year ago

madhavdube commented 1 year ago

I am trying to set the default args as default_args={'on_failure_callback': 'foo_function_str`}

the yaml config I am trying

dag_1:
   default_args:
       start_date: '1999-01-01'
       on_failure_callback: 'dags.utils.failures.send_alert' -- str representation of function

The other config I tried using on_failure_callback_file but this does not set the property in the default dict.

dag factory version - 0.8.0

subbota19 commented 2 months ago

Hi @madhavdube,

I tried creating the DAG based on your config file using the latest version of dag-factory v0.19.0, and it worked fine. Please try running it with the latest version.

As for specifying callbacks, another method is to use the parameters: _on_failure_callbackfile and _on_failure_callbackname. You can find more details in the documentation.

One limitation is that currently it's not possible to provide callbacks within _defaultargs. You can only define them at the DAG level, like this

dag_test:
  default_args:
    start_date: '2024-09-14'
  on_success_callback_file: "/usr/local/airflow/dags/utils/utils/failures.py"
  on_success_callback_name: "send_alert"

However, I've created a PR that will allow specifying callbacks inside _defaultargs, enabling callback configurations for individual DAG tasks as well: https://github.com/astronomer/dag-factory/pull/218.