search

astronomer / dag-factory

Dynamically generate Apache Airflow DAGs from YAML configuration files
Apache License 2.0
1.19k stars 179 forks source link

execution_delta and execution_date_fn attribute in external_task_sensor #226

Open atuweetseng opened 1 month ago

atuweetseng commented 1 month ago

Hi experts,

I tried to use the external_task_sensor in dag yaml but I can make either one of the execution_delta and execution_date_fn attribute works. For the execution_date_fn attribute I wrote a customized function: get_exec_date_fn and put it in the dag loader python code by calling load_yaml_dags method. And in the yaml I configured as the below: Wait_LOADING_Job: operator: airflow.sensors.external_task_sensor.ExternalTaskSensor external_dag_id: "DS_BP" external_task_id: "BP_LOADING" execution_date_fn: get_exec_date_fn timeout: 3 retries: 0 mode: reschedule dependencies: [ start_job ] But I always got the "TypeError: 'get_exec_date_fn' is not a callable object"

And then I tried to use execution_delta attribute with "days: 1" or "1 days" I always got: TypeError: unsupported operand type(s) for -: 'DateTime' and 'dict'/'str'

My question is what is the correct representation of execution_delta value? And, how to configure the execution_date_fn in a correct way?

Thank you!

klainedl commented 2 weeks ago

For execution_date_fn attribute , I managed to make this work using parameters execution_date_fn_name and execution_date_fn_file. Reference: https://github.com/astronomer/dag-factory/blob/main/dagfactory/dagbuilder.py#L458

atuweetseng commented 2 weeks ago

And for the execution_delta, it seems can use the execution_delta_secs instead. But for the execution_date_fn_name and execution_date_fn_file, do you have any suggestion that I can set arguments for the fn function? It seems there is no additional parameters setting can support it. Thank you!

tatiana commented 1 week ago

Hi @atuweetseng would you like to add this feature to DAG Factory? We'd love contributions