PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.66k stars 1.53k forks source link

Per-schedule parameters for worker-based deployment schedules #14524

Open muru opened 1 month ago

muru commented 1 month ago

First check

Prefect Version

2.x

Describe the current behavior

We're using worker-based deployments. AFAICT from https://docs.prefect.io/latest/concepts/schedules/ and https://docs.prefect.io/latest/concepts/deployments/, parameters can only be set for a deployment as a whole. I suppose this is moot given this caveat:

Support for multiple schedules in flow.serve, flow.deploy, serve, and worker-based deployments with prefect deploy will arrive soon.

Describe the proposed behavior

If/when support for multiple schedules arrives, we'd like to have different schedules each with its own set of parameters in a given deployment.

Example Use

Our current use case would be a clean-up flow which is parametrized over services, and we'd like to have a single deployment scheduling the clean-up for a bunch of them, but via custom schedules for each service to avoid cluttering up the deployments list.

Additional context

Apparently this used to be possible for Prefect v1, if I read this discussion right: https://github.com/PrefectHQ/prefect/discussions/4543.

TWeatherston commented 1 month ago

Big +1 from us for this idea. We used it quite a bit in V1 and it would be very useful for selectively running certain DBT models on a schedule

williamjamir commented 1 month ago

I'm also looking for a way to have one deployment with multiple schedules, and each schedule has different parameter values.

My use case is for ML pipelines.

TheoBabilon commented 1 month ago

Same here, definitely something we'd be interested in!

Sohambutala commented 1 month ago

Much needed feature! Is there any simpler workaround for this, besides deploying multiple flows or handling array of inputs into a single deployment?

delkirksey commented 1 week ago

+1 here! Would be an extremely useful feature.