apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.88k stars 4.63k forks source link

[DSIP-55][Master] Separate the waiting dispatched task into different queue by worker group #16260

Open ruanwenjun opened 4 months ago

ruanwenjun commented 4 months ago

Search before asking

Motivation

Right now, in master DS will use one single thread to dispatch the tasks. The problem is all tasks are stored at one waiting queue, if a task dispatch failed then it will be moved to the end of waiting queue.

There exist a case, if a worker is overload at one worker group, and A, B ,C are using the worker, the priority is A > B > C, if the A dispatched failed, then begin to dispatch B, but the worker changed from overload to normal, then B will be dispatched successfully, this will caused problem, since priority of A is higher than B.

Design Detail

Different worker group use different waiting queue.

image

We need to make one waiting queue should only be handled by one dispatch thread. Once a task dispatched failed it should be put back to its worker group queue.

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

No response

Code of Conduct