search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.1k stars 3.56k forks source link

Unload Rate Limiting during Graceful Shutdown #20753

Open labuladong opened 1 year ago

labuladong commented 1 year ago

Search before asking

Motivation

When Pulsar undergoes a graceful shutdown, it unloads all topics, but there is no limit on the unloading rate. As a result, during a rolling upgrade, the broker cluster might generate a large volume of lookup requests in a short period, which could affect the service of the broker. Implementing a limit on the unload rate would make the upgrade process smoother.

Solution

This issue has been previously raised in a pull request https://github.com/apache/pulsar/pull/14114 and a solution was proposed by adding an unload interface with concurrency control for admin CLI.

However, the current broker graceful shutdown doesn't use this interface. Hence, my proposal is pretty straightforward:

  1. Add a dynamic config named brokerShutdownMaxBundleUnloadPerMinute to the Pulsar broker to control the concurrency of unloading.

  2. Modify this code to use this function with the concurrency parameter for unloading.

Alternatives

No response

Anything else?

mail list: https://lists.apache.org/thread/7f5f9gn7qq86hstrkw5oz51g0rxrqq37

Are you willing to submit a PR?

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.