search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.66k stars 1.65k forks source link

Prefect All Flow runs stay scheduled while not start running #15042

Open Yannlihan1208 opened 3 months ago

Yannlihan1208 commented 3 months ago

Bug summary

After I started my deployment flow in my own prefect server, all flows stay state "scheduled" while none of them start running. I have tried restarting my agent but still not work. image

Version info (prefect version output)

Version:             2.19.7
API version:         0.8.4
Python version:      3.8.10
Git commit:          60f05122
Built:               Fri, Jun 28, 2024 11:27 AM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         server

Additional context

Here is my worker pool image

Yannlihan1208 commented 3 months ago

Actually I am using dask cluster for running, and dask hasn't received anything. Here is the agent log: image

Yannlihan1208 commented 3 months ago

By the way, I cancelled about 100 flow runs before I created these flow runs. I cancelled by calling the Prefect REST API to set the flow run state to "CANCELLED". Not sure if this REST API truly cancelled flow runs. image

Yannlihan1208 commented 3 months ago

I think it's the problem of concurrency limit of run pnl batch work queue. Because other flows from other queues work properly. But I can not see any flow run in run pnl batch work queue. Maybe just calling REST API to set state or delete running flows in Prefect UI didn't truly remove the flow run from loop event?