[POP] Make lite pull consumer support POP mode to consume

[cserwen]

FEATURE REQUEST

1. Please describe the feature you are requesting.

   • Support pop mode for DefaultLitePullConsumer.

2. Provide any additional detail on your proposed use case for this feature.

• Currently, POP mode is only supported in Push consumer. However, in the push consumption mode, users can only passively accept messages to consume, and cannot actively control the speed of consumption (the function of limited flow is not very flexible).
• As we thought, pull consumer can achieve this purpose very well. However, in some business scenarios, there may be thousands of instances in the consumer group (which will automatically expand and shrink with traffic). If the client-side rebalance mode is still used, it will lead to unbalanced consumption traffic and frequent rebalance. So it is necessary to make pull consumption to support POP mode.
• Possible code looks like this:

  public static void main(String[] args) throws Exception {
    DefaultLitePullConsumer litePullConsumer = new DefaultLitePullConsumer("lite_pull_consumer_test");
    litePullConsumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
    litePullConsumer.subscribe("pop-topic", "*");  //POP consumption only takes effect in subscription mode
    litePullConsumer.setNamesrvAddr("localhost:9876");
    litePullConsumer.setClientRebalance(false);
    litePullConsumer.start();
    try {
        while (running) {
            List<MessageExt> messageExts = litePullConsumer.poll();
            for (MessageExt messageExt : messageExts) {
                System.out.printf("%s=%s\n", messageExts.size(), messageExts);
                litePullConsumer.ack(messageExt);                   
  //                litePullConsumer.changePopInvisibleTime(messageExt);   //retry
            }
        }
    } finally {
        litePullConsumer.shutdown();
    }
  }

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

• should-have

[zhouxinyu]

Hi, There is no need to do that, lite pull is widely used in the streaming scenario while POP is unsuitable.

For your needs, I think the new SimpleConsumer API is more suitable and could better leverage the POP ability. Please refer to here for more information.

[cserwen]

Hi, There is no need to do that, lite pull is widely used in the streaming scenario while POP is unsuitable.

For your needs, I think the new SimpleConsumer API is more suitable and could better leverage the POP ability. Please refer to here for more information.

@zhouxinyu Got it. But it seems that it can only be used after 5.0. Looking forward to the 5.0 version.

[ferrirW]

Hi, There is no need to do that, lite pull is widely used in the streaming scenario while POP is unsuitable.

For your needs, I think the new SimpleConsumer API is more suitable and could better leverage the POP ability. Please refer to here for more information.

In our case, the user has 500 machines and needs to explicitly call the receive interface for logic processing.

Can SimpleConsumer achieve this effect, similar to pop ? @zhouxinyu

[zhouxinyu]

Hi, There is no need to do that, lite pull is widely used in the streaming scenario while POP is unsuitable. For your needs, I think the new SimpleConsumer API is more suitable and could better leverage the POP ability. Please refer to here for more information.

In our case, the user has 500 machines and needs to explicitly call the receive interface for logic processing.

Can SimpleConsumer achieve this effect, similar to pop ? @zhouxinyu

Partly Yes, SimpleConsumer will receive messages from a proxy, usually, it's an LB, then the proxy will pop messages from the 500 machines in a round-robin manner. It's obvious that the clients don't need to send requests to all backend servers but should improve concurrency to reduce false empty response. In the future, the proxy could long-polling from all the backend machines to reduce client pressure.

[zhouxinyu]

BTW, do you use the POP technology in your production environment? If yes, any feedback?

[cserwen]

BTW, do you use the POP technology in your production environment? If yes, any feedback?

@zhouxinyu Yes, we are already using the POP consumption function, and through some optimizations (flexible delay messages instead of level delay), it has shown a good use effect. But there is currently such a scenario that the pull consumer needs to support the POP mode.

[zhouxinyu]

Got it, please pay some attention to SimpleConsumer and the implementation of server-side: https://github.com/apache/rocketmq/pull/4225