Support user defined and pluggable consumer selector

[isminex]

Is your enhancement request related to a problem? Please describe. When we try to Introduce pulsar to our existing system, compatibility is our first consideration. Firstly, our production system uses jump consistent hash algorithm to select consumers, but pulsar uses range consistent hash algorithm natively. It's impossible for us to guarantee that the same node can consume data with the same routing key from pulsar and our existing system simultaneously. Secondly, our system requires no change in consumer selecting rules when one or more consumers disconnect temporarily(service upgrade.etc). As our service is locally stateful and will be updated frequently, constantly route-jitter is harmful for us.

Describe the solution you'd like Support user defined and pluggable consumer selector, so we can manage consumers' behavior by ourself when dispath messages.

Describe alternatives you've considered We have implemented the same routing algorithm and consumer management strategy as our existing system in broker. But this implementation will cause great trouble for us to update the version of puslar regularly in the future. @codelipenghui

[gaozhangmin]

Consumer selector is only used in Key_Shared mode. Right？ May be we can implement it like MessageRouter.

[gaozhangmin]

I can help implement it.

[isminex]

Consumer selector is only used in Key_Shared mode. Right？ May be we can implement it like MessageRouter.

Yes, we just use the Key_Shared mode now.

[isminex]

I can help implement it.

Nice to hear that! Can I have your implementation process and schedules？

[gaozhangmin]

@isminex what you need is config selector class through broker configuration, Right?

[isminex]

@isminex what you need is config selector class through broker configuration, Right?

Yes, that's it.

[gaozhangmin]

I will implement it this week.

[gaozhangmin]

@isminex A pr is committed. PTAL

[isminex]

PTAL

LGTM. Thanks very much.

[eolivelli]

Maintaining such extensions will be very hard for users as we are deep inside the internals of Pulsar.

We should define a smaller surface for the extension point, like extracting the only part that selects the consumers

[gaozhangmin]

@isminex After discuss with community, We don't support this PIP， the exists consumer selector strategy is enough for most Scenes. If we allow users to customize this strategy, it may lead to a misuse of key shared subscription which doesn't conform to the intention of origin design of key shared subscription.

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.

[andyzhao9527]

@isminex After discuss with community, We don't support this PIP， the exists consumer selector strategy is enough for most Scenes. If we allow users to customize this strategy, it may lead to a misuse of key shared subscription which doesn't conform to the intention of origin design of key shared subscription.

@gaozhangmin It's too hard to see this news. We also encounter the problem of consumer routing compatibility in the legacy system upgrade. If the community can't support this feature, is there any better suggestion for the legacy system routing compatibility problem? At present, we can think of two ways. One is custom development, but it's troublesome to synchronize the subsequent community version upgrades; The other is that we add a pulsar_ Adapt. This service ensures the routing consistency with the old system, but this brings additional machine costs.

[eolivelli]

I understand your problem. I have recently worked on Broker Side Filtering and this feature is complementary.

I can't remember any pointers about the discussion. Would you please start (or restart?) a discussion on dev@pulsar.apache.org ? not everyone follows GH issues

[andyzhao9527]

I understand your problem. I have recently worked on Broker Side Filtering and this feature is complementary.

I can't remember any pointers about the discussion. Would you please start (or restart?) a discussion on dev@pulsar.apache.org ? not everyone follows GH issues

Thank you for your attention. This issue was discussed in this issue: https://github.com/apache/pulsar/issues/13347 And gaozhangmin has also completed the development of features: https://github.com/apache/pulsar/issues/13473 However, this feature was rejected in the merger vote.You also gave the corresponding reasons for rejection. I don't know whether community has new considerations to support this feature, If community can reconsider and support this feature, it will bring great convenience to our project. No matter what the result is, Thank you first.