search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
35.3k stars 13.78k forks source link

Deferrable HttpSensor does not move to Triggerer when using `response_check` #40209

Open josh-fell opened 3 weeks ago

josh-fell commented 3 weeks ago

Discussed in https://github.com/apache/airflow/discussions/39597

Originally posted by **bejota** May 13, 2024 Hello, I'm using HttpSensor to poll SLURM's REST API for job status. I'm using `deferrable` mode with `response_check`. The task only goes to the triggerer if I remove the `response_check`. Airflow Docker Image `apache/airflow:latest-python3.9` Airflow Version `2.9.0` apache-airflow-providers-http `4.10.0` Does not move to triggerer: ``` check_slurm_job = HttpSensor( task_id="check_slurm_job", http_conn_id="slurm_conn_id", method="get", endpoint="slurm/v0.0.40/job/999", response_check=lambda response: "COMPLETED" in response.text, poke_interval=10, timeout=60 * 60 * 5, deferrable=True, # not deferring to triggerer ? dag=dag, ) ``` Moves to triggerer: ``` check_slurm_job = HttpSensor( task_id="check_slurm_job", http_conn_id="slurm_conn_id", method="get", endpoint="slurm/v0.0.40/job/999", # response_check=lambda response: "COMPLETED" in response.text, poke_interval=10, timeout=60 * 60 * 5, deferrable=True, # not deferring to triggerer ? dag=dag, ) ``` In digging further, I see that HttpTrigger has no `response_check` method. I will attempt to add one, unless there is a better approach to this problem?
josh-fell commented 2 weeks ago

Probably what the solution should be. May need to confirm with HttpOperator too:

https://github.com/apache/airflow/discussions/39597#discussioncomment-9758322