PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.64k stars 1.65k forks source link

"Transactional" and test deployments #12215

Open nzojy opened 8 months ago

nzojy commented 8 months ago

First check

Prefect Version

2.x

Describe the current behavior

hey all! we are running prefect deploy --all in our CI/CD. sometimes a few Flows will succeed to deploy to the cloud, then a deployment will fail because of something like a pydantic error and topple over our CI/CD. At that point the state of the deployments in the cloud is out of whack.

Describe the proposed behavior

Ideally there's a way to make make deployments transactional where they are all or nothing.

Also, if we could mock or dry run deployments so we can test whether they succeed before we actually deploy them to our cloud instance, that would be super helpful here!

Example Use

No response

Additional context

No response

jakekaplan commented 8 months ago

Thanks for filing this enhancement request @nzojy!

Wanted to clarify that when you say "out of wack", you mean some of your deployments are applied but not all because they are sequentially updated? So if you have 5 deployments and the 3rd one fails to update, 2 are updated and 3 are not? Is that correct?

YoungiiJC commented 8 months ago

@jakekaplan -- this is @nzojy commenting from my personal github account.

if you have 5 deployments and the 3rd one fails to update, 2 are updated and 3 are not? Is that correct?

yes, exactly