astronomer / dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Apache License 2.0
1.16k stars 176 forks source link

Add support for templating in `on_failure_callback` #209

Open jroach-astronomer opened 1 month ago

jroach-astronomer commented 1 month ago

In reference to https://github.com/astronomer/dag-factory/issues/206, add support for templating in on_failure_callback. Requester is specifically looking to support templated parameters that do not need to be stored in a .py file.

Here's an example (from the issue submitted by @matveykortsev) for reference:

from airflow.providers.slack.notifications.slack import send_slack_notification
'on_failure_callback': [
        send_slack_notification(
            slack_conn_id='slack',
            text="""
                :red_circle: Task Failed. 
                *Task*: {{ ti.task_id }}  
                *Dag*: {{ ti.dag_id }} 
                *Execution Time*: {{ ti.execution_date }}  
                *Log Url*: {{ ti.log_url }} 
                """,
            channel="analytics-alerts",
            username="Airflow",
        )
    ],

@pankajastro has created an issue (https://github.com/astronomer/dag-factory/issues/200) to ensure consistency between native Airflow and dag-factory.

jroach-astronomer commented 1 month ago

This would be a new feature, I'm going to see if it's something that's feasible. Basically, we'd need to support functionality that looks like this:

<dag-id>:
  default_args:
    on_failure_callback: <path-to-function>
      <arg-name>: value
      <arg-name>: value