Closed heesung-sn closed 1 year ago
I think the part of More aggressive load balance strategy
mentioned in this proposal is resolved by https://github.com/apache/pulsar/pull/17456
For the load balance epochs. I'm not if I understand it correctly. Is it will expose metrics or something to help to tune the max_transfer_cnt
?
Hi,
Regarding More aggressive load balance strategy
, I think the idea here
is similar from #17456 https://github.com/apache/pulsar/pull/17456.
Even if no global shedding condition is met (here the shedding condition is
std-based instead of avg threshold), and if the min resource usage broker
is having no traffic, this pip is still trying to transfer load from max to
min broker.
Yes. This pip proposes standard deviation to model the load balance optimization problem. There will be metrics(e.g. bundle_transfer_epoch, bundle_transfer_count, broker_load_std, broker_load_avg) to show how many bundle_transfer_count(epochs) are taken to reach the target(target_std). (bundle_transfer_epoch is reset to zero once the running std reaches target_std). In this way, we can clearly monitor the time(epochs) and bundle_transfer_count to reach the target load distribution. Accordingly, one could tune the load balance configs like max_transfer_cnt you mentioned.
Regards, Heesung
On Sun, Oct 30, 2022 at 7:02 PM Penghui Li @.***> wrote:
I think the part of More aggressive load balance strategy mentioned in this proposal is resolved by #17456 https://github.com/apache/pulsar/pull/17456
For the load balance epochs. I'm not if I understand it correctly. Is it will expose metrics or something to help to tune the max_transfer_cnt?
— Reply to this email directly, view it on GitHub https://github.com/apache/pulsar/issues/18215#issuecomment-1296442299, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYVJ675NMQQHDTTAVPLRUXTWF4SDBANCNFSM6AAAAAARPSHN6Q . You are receiving this because you authored the thread.Message ID: @.***>
Hi,
I will raise a vote soon if there are no more questions or comments.
Thanks, Heesung
On Mon, Oct 31, 2022 at 10:19 AM Heesung Sohn @.***> wrote:
Hi,
Regarding
More aggressive load balance strategy
, I think the idea here is similar from #17456 https://github.com/apache/pulsar/pull/17456. Even if no global shedding condition is met (here the shedding condition is std-based instead of avg threshold), and if the min resource usage broker is having no traffic, this pip is still trying to transfer load from max to min broker.Yes. This pip proposes standard deviation to model the load balance optimization problem. There will be metrics(e.g. bundle_transfer_epoch, bundle_transfer_count, broker_load_std, broker_load_avg) to show how many bundle_transfer_count(epochs) are taken to reach the target(target_std). (bundle_transfer_epoch is reset to zero once the running std reaches target_std). In this way, we can clearly monitor the time(epochs) and bundle_transfer_count to reach the target load distribution. Accordingly, one could tune the load balance configs like max_transfer_cnt you mentioned.
Regards, Heesung
On Sun, Oct 30, 2022 at 7:02 PM Penghui Li @.***> wrote:
I think the part of More aggressive load balance strategy mentioned in this proposal is resolved by #17456 https://github.com/apache/pulsar/pull/17456
For the load balance epochs. I'm not if I understand it correctly. Is it will expose metrics or something to help to tune the max_transfer_cnt?
— Reply to this email directly, view it on GitHub https://github.com/apache/pulsar/issues/18215#issuecomment-1296442299, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYVJ675NMQQHDTTAVPLRUXTWF4SDBANCNFSM6AAAAAARPSHN6Q . You are receiving this because you authored the thread.Message ID: @.***>
discussion email thread: https://lists.apache.org/thread/9k4968h57ffc6q2g6zn1tnbz5ql234x7 vote email thread: https://lists.apache.org/thread/cwnyjcy8vw9mb08jwohn9l1d3g62wtd1
Raised a PR : https://github.com/apache/pulsar/pull/18865
Note: we will have separate PIP and PRs to define the new broker load balancer metrics centrally.
The issue had no activity for 30 days, mark with Stale label.
Motivation
bundle transfer protocol
, we could implement a new shedding strategy to specify a new owner broker.More aggressive load balance strategy: https://docs.google.com/document/d/1nXaiSK39E10awqinnDUX5Sial79FSTczFCmMdc9q8o8/ (reported) We need to revisit the current load balance strategy to balance the load more evenly and frequently. With the current static threshold approach, sometimes it appears that the load balance is not working when 1. the threshold is set too high, and overall usage is relatively low, and 2 when new brokers are added.
Repeated Shedding due to highly-weighted historical load: https://github.com/apache/pulsar/issues/18173 (reported)
Goal
bundle transfer
protocol introduced in https://github.com/apache/pulsar/issues/16691API Changes
No.
Implementation
Pseudo code
The idea is straightforward. We want to keep unloading bundles from max loaded broker to min loaded broker until the standard deviation of the broker load distribution is below our target.
The following is the Pseudo code.
Theoretical Load Balance Epochs(Transfer Counts) per Cluster Size for Target std=15
This data shows how many
epochs
are required to tune themax_transfer_cnt
config.In general, the number of required transfer cycles(minutes) for (target std=15) =
epochs
/max_transfer_cnt
Alternatives
N/A
Anything else?
No response