PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.6k stars 1.65k forks source link

Deploy all to enable sync of yaml and server deployments #15332

Open frantakalab opened 2 months ago

frantakalab commented 2 months ago

Describe the current behavior

Assume I have two deployments in my prefect.yaml:

name: importers

deployments:
- name: deployment-1
  entrypoint: one.py:flow

- name: deployment-2
  entrypoint: two.py:flow

I run prefect --no-prompt deploy --all --prefect-file prefect.yaml and two of these deployments get created on the server.

Next I delete deployment-2 in my yaml. I run the same deploy command, on the server deployment-1 is updated and deployment-2 is kept.

Describe the proposed behavior

The command should synchronize the definitions in the yaml with the server (check for deletion as well), so deployment-2 would be deleted. If someone is needing this "non-deletion" behavior, then adding a flag like --sync, or --sync-all would seem like a good descriptive name.

Important is to keep this per name, to make sure this does not delete deployments defined and created from other places.

Example Use

for automated deployments, where deployment definitions are as a code I would expect synchronisation to be the default.

Additional context

I have checked with Marvin before posting here: https://prefect-community.slack.com/archives/C04DZJC94DC/p1726045580886509

fchareyr commented 1 week ago

This would be especially useful given that there is currently no prefect deployment delete --all that could be used in a CI/CD to delete all deployments before (re-)deploying the ones specified in prefect.yml.

frantakalab commented 1 week ago

This would be especially useful given that there is currently no prefect deployment delete --all that could be used in a CI/CD to delete all deployments before (re-)deploying the ones specified in prefect.yml.

Thanks for the support, the two step process -- delete/deploy would also be an option, but still I think some cases need proper handling, for example, deleting a deployment of a running flow, or redeploying of a critical and frequent flow without causing any outage or skipped flow runs.