apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.12k stars 3.57k forks source link

[Bug] Can't split bundle: The size of split boundaries is not 1 - when using flow_or_qps_equally_divide #22254

Open Bloodyludi opened 6 months ago

Bloodyludi commented 6 months ago

Search before asking

Version

v3.2.0

Minimal reproduce step

Using the flow_or_qps_equally_divide bundle split algorithm with following config:

defaultNumberOfNamespaceBundles: "9"
defaultNamespaceBundleSplitAlgorithm: "flow_or_qps_equally_divide"
loadBalancerNamespaceBundleMaxMsgRate: "250"

Bundles are not split when multiple splitBoundary are returned by the algorithm: https://github.com/apache/pulsar/blob/v3.2.0/pulsar-broker/src/main/java/org/apache/pulsar/common/naming/FlowOrQpsEquallyDivideBundleSplitAlgorithm.java#L50-L128

2024-03-05T12:56:26,943+0000 [pulsar-load-manager-1-1] INFO  org.apache.pulsar.broker.loadbalance.extensions.strategy.DefaultNamespaceBundleSplitStrategyImpl - Splitting bundle: pinger/v2.0.0/0xaaaaaaa8_0xc71c71c4. Topics: 54/1000, Sessions: (125+51)/1000, Message Rate: 2077.90/250 (msgs/s), Message Throughput: 2.19/100 (MB/s)
2024-03-05T12:56:26,945+0000 [pulsar-load-manager-1-1] WARN  org.apache.pulsar.broker.loadbalance.extensions.strategy.DefaultNamespaceBundleSplitStrategyImpl - Can't split broker:pinger/v2.0.0/0xaaaaaaa8_0xc71c71c4. The size of split boundaries is not 1. splitBoundary:[3211714940, 3266565899]

What did you expect to see?

The bundle is split into multiple bundles at the returned boundaries.

What did you see instead?

Bundles are not split at all and the above mentioned warning is logged by this condition: https://github.com/apache/pulsar/blob/v3.2.0/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/strategy/DefaultNamespaceBundleSplitStrategyImpl.java#L255-L260 It looks like the actual splitting code would work even with multiple split boundaries returned, so I'm wondering why this condition is there in the first place?

Anything else?

I was able to execute the split manually by running:

bin/pulsar-admin namespaces split-bundle --bundle 0xaaaaaaa8_0xc71c71c4 --split-boundaries 3169176886 -san specified_positions_divide <namespace>

After this, subsequent splits were done automatically again.

Are you willing to submit a PR?

lhotari commented 6 months ago

@lordcheng10 any chance to check this issue? thanks