apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.36k stars 14.1k forks source link

deferrable FileSensor alway get stuck #41087

Open WiC-htao opened 1 month ago

WiC-htao commented 1 month ago

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.9.1

What happened?

when i use FileSensor and set deferrable=True,i got the warning, “The triggers does not appear to be running. Triggers will not run, and any deferred operator will remian deferred until it times out and fails”, and the task looks stuck. But everything is good when i set deferrable =False.

I am not sure if this is a bug or I mis-config anything. My python version is 3.8.10 with airflow 2.9.1 using a mysql as a dataset backend on ubuntu 20.04. And code almost running on gpfs file system. I don‘t know whether it doesn't support deferrable operators and cause all of this.

My test code is really simple here.

with DAG("dev_test", schedule=None) as dag:
      FileSensor(task_id = "test", filepath=“/home/ubuntu/test.airflow”, deferrable=True, poke_interval=30, timeout=300,

The only warning message i can find is

WARNING - empty cryptography key - values will not be stored encrypted.

by the way, airflow db clean also doesn't work for me with a warning below, i have to delete the trigger table mannual in sql.

WARNING - Encountered error when attempting to clean table 'trigger'

What you think should happen instead?

No response

How to reproduce

with DAG("dev_test", schedule=None) as dag:
      FileSensor(task_id = "test", filepath=“/home/ubuntu/test.airflow”, deferrable=True, poke_interval=30, timeout=300,

And trigger it mannually.

Operating System

ubuntu 20.04

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 1 month ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.