[CELEBORN-1636][FOLLOWUP] Dynamic resources will only be utilized in case of candidates shortages

apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

https://celeborn.apache.org/

Apache License 2.0

886 stars 359 forks source link

[CELEBORN-1636][FOLLOWUP] Dynamic resources will only be utilized in case of candidates shortages #2852

Closed zaynt4606 closed 1 day ago

zaynt4606 commented 1 week ago

What changes were proposed in this pull request?

Follow up of [https://github.com/apache/celeborn/pull/2835] Only use dynamic resources when candidates are not enough. And change the way geting availableWorkers form heartbeat to requestSlots RPC to avoid the burden of heartbeat.

Why are the changes needed?

Does this PR introduce any user-facing change?

Add another configuration.

How was this patch tested?

zaynt4606 commented 5 days ago

cc @FMX @RexXiong

RexXiong commented 1 day ago

Thanks, merge to main(v0.6.0)