PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.82k stars 1.55k forks source link

Deploy flows by matching paths #10270

Open EmilRex opened 1 year ago

EmilRex commented 1 year ago

First check

Prefect Version

2.x

Describe the current behavior

The prefect deploy command can create deployments defined in a prefect.yaml in two ways:

  1. By referring to one or more deployments by name: prefect deploy -n flow-a/default -n flow-b/default
  2. With the --all flag: prefect deploy --all

Describe the proposed behavior

When deploying via something like GitHub actions, users often want to deploy only flows in files that have changed. Being able to specify a glob-style pattern on matching files would make this workflow more convenient. One possible implementation might be to create a -m / --match argument that matches deployments based on a file pattern.

Example Use

Suppose you have something like the following prefect.yaml file:

deployments:
- name: default
  entrypoint: flows/extract/source_a.py:main
- name: default
  entrypoint: flows/extract/source_b.py:main
- name: default
  entrypoint: flows/load/destination_a.py:main

Running this command:

prefect deploy -m "flows/extract/*"

Should result in deploying the first two flows.

Additional context

This could perhaps also be -f / --file or similar, though -f is occupied by --flow. There are probably better implementations, but the main point is deploying flows already declared in prefect.yaml based on glob-style paths.

billpalombi commented 1 year ago

Great idea