search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
16.33k stars 1.59k forks source link

Deleting a deployment does not delete corresponding scheduled runs #15697

Open 0leg-st opened 2 weeks ago

0leg-st commented 2 weeks ago

Bug summary

Deleting a deployment does not always delete all the scheduled runs that correspond to this deployment. If runs are not deleted, the system tries to execute them and falls into an infinite loop once the deployment is not found. Problem occurs 1 time in 10 without apparent reason.

Version info (prefect version output)

Version:             3.0.0
API version:         0.8.4
Python version:      3.12.5
Git commit:          c40d069d
Built:               Tue, Sep 3, 2024 11:13 AM
OS/Arch:             linux/aarch64
Profile:             ephemeral
Server type:         server
Pydantic version:    2.8.2

Additional context

No response

mvdb-enspi commented 2 weeks ago

Noticed the same on Prefect 3.0.3. 

Version:             3.0.3
API version:         0.8.4
Python version:      3.10.11
Git commit:          d9c51bc3
Built:               Fri, Sep 20, 2024 8:52 AM
OS/Arch:             linux/x86_64
Profile:             local
Server type:         server
Pydantic version:    2.9.2
Integrations:
  prefect-docker:    0.6.1
  prefect-aws:       0.5.0

Like you said, it doesn't happen consistently but once so often. In the meantime, is there an easy way to delete scheduled runs with a non-existent deployment?

mvdb-enspi commented 2 weeks ago

To give some more context, there are no error logs for the Prefect server and I use Postgres as the database.

0leg-st commented 4 days ago

Hello! Any news about the matter or additional infos you could give us ? Is this indeed a bug you're aware of or are we missing something on our side?

mvdb-enspi commented 4 days ago

@0leg-st I'm not 100% sure, since I didn't experiment with it, but I suspect the issue is caused when the new deployment somehow is no longer connected to the old one. In my case it happened because I changed the DockerImage name in the deploy function.