PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.66k stars 1.65k forks source link

Allow workers to configure work-queue priorities #13826

Open GoldToothRichards opened 6 months ago

GoldToothRichards commented 6 months ago

First check

Prefect Version

2.x

Describe the current behavior

Work-Queue Behavior

Each prefect work-pool can have any number of work-queues associated to it. Each work-queue can be assigned an integer priority by using the --priority flag in the prefect CLI:

prefect work-queue create my-queue --pool my-pool --priority 1

Worker Behavior

When starting a prefect worker, you can select one or more work-queues that the worker should pull work from by using the --work-queue flag in the prefect CLI:

prefect worker start --name my-worker --pool my-pool --work-queue my-queue

Describe the proposed behavior

Worker Behavior

When starting a prefect worker, you can select one or more work-queues that the worker should pull work from together with custom priorities. I am hoping for something along the lines of:

prefect worker start --name my-worker --pool my-pool --work-queue-priority-config "{'queue-1': 1, 'queue-2': 2}"

Example Use

This enhancement would allow different workers to poll a single work-pool but prioritize the work differently.

As an example, our current use case is this:

We would like to run one worker on each of our GPU servers and have all of those workers accept jobs from a common "gpu_jobs" work-pool. Within the gpu_jobs pool, we would like to divide the jobs into different work-queues depending on where the file is actually located (i.e. one work-queue for each server). We would like each worker to give the highest priority to it's own work-queue, but accept any other jobs with a lower priority.

For example, we would like to do:

Additional context

No response

desertaxle commented 5 months ago

Thanks for the enhancement request @GoldToothRichards! This will require some design on our end to determine the feasibility, but we'll respond here once we have a potential design.