apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.12k stars 14.31k forks source link

on_dag_run_created listener ignored in plugin #44071

Open WillRaphaelson opened 2 hours ago

WillRaphaelson commented 2 hours ago

Apache Airflow version

2.10.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

on_dag_run_created hook is being ignored in plugin

What you think should happen instead?

on_dag_run_created should run the relevant logic in the function

How to reproduce

from airflow.plugins_manager import AirflowPlugin
from airflow.listeners import hookimpl
import logging

# Configure logger
logger = logging.getLogger(__name__)

class PreDagRunListener:
    """
    Listener to perform actions before any task in a DAG run starts.
    """

    @hookimpl
    def on_dag_run_created(self, dag_run, session):
        """
        Hook triggered when a DAG run is created, before tasks are executed.
        """
        dag_id = dag_run.dag_id
        run_id = dag_run.run_id

        logger.info(f"Pre-DAG execution logic triggered for DAG {dag_id}, run_id {run_id}.")

        # Add your custom pre-DAG logic here
        # Example: Block a specific DAG
        if dag_id == "blocked_dag":
            logger.error(f"DAG {dag_id} is blocked from running.")
            raise ValueError(f"DAG {dag_id} failed pre-run checks.")

        logger.info(f"DAG {dag_id} passed pre-run checks.")

class PreDagRunPlugin(AirflowPlugin):
    name = "pre_dag_run_plugin"
    listeners = [PreDagRunListener()]

Operating System

MacOS

Versions of Apache Airflow Providers

apache-airflow-providers-common-compat==1.2.1 apache-airflow-providers-common-io==1.4.2 apache-airflow-providers-common-sql==1.19.0 apache-airflow-providers-fab==1.5.0 apache-airflow-providers-ftp==3.11.1 apache-airflow-providers-http==4.13.2 apache-airflow-providers-imap==3.7.0 apache-airflow-providers-smtp==1.8.0 apache-airflow-providers-sqlite==3.9.0

Deployment

Virtualenv installation

Deployment details

Local setup - only change i made was to not lazy load plugins with lazy_load_plugins = False

Anything else?

I can never get an on_dag_run_created hook to run no matter what. I'm open ot anything that allows me to run dag level pre-flight checks.

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 2 hours ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.