search

apache / pulsar-dotpulsar

The official .NET client library for Apache Pulsar
https://pulsar.apache.org/
Apache License 2.0
234 stars 60 forks source link

Consumer with MaxDegreeOfParallelism eventually hangs #223

Closed smbecker closed 2 months ago

smbecker commented 3 months ago

Description

I have a batch process that when it kicks off, will fill a topic with around 100k messages in a very short period of time. I have a series of consumers that start pulling from that topic. If I use the default MaxDegreeOfParallelism=1, then everything finishes successfully as expected. I wanted to play around with how changes to MaxDegreeOfParallelism impact performance. My first test was using MaxDegreeOfParallelism=10. Unfortunately, after a few thousand messages, it just stopped consuming. I am still trying to work through where it is hung up. If I restart the process, it will pick back up and get through another few thousand before getting stuck again. Any idea where to start debugging this issue? I am using MessagePrefetchCount=1 and SubscriptionType=KeyShared for what its worth.

Reproduction Steps

In my case, having a topic with tens of thousands of messages in backlog and trying to process it with MaxDegreeOfParallelism=10.

Expected behavior

It should process the messages as expected

Actual behavior

It stops processing messages after a few thousand or so

Regression?

No response

Known Workarounds

Change MaxDegreeOfParallelism=1

Configuration

No response

Other information

No response

smbecker commented 3 months ago

I think that I found my issue here. It seemed that the consumer was getting stuck in Receive. When looking at the broker stats, it showed that the consumer subscription was blockedConsumerOnUnackedMsgs and that there was a pretty large number of unackedMessages. After some more digging, I found a note here that mentioned that cumulative acknowledgements cannot be used in shared or key_shared subscriptions, which I was using. This would never become an issue with MaxDegreeOfParallelism=1. Perhaps an exception should be thrown if MaxDegreeOfParallelism > 1 && EnsureOrderedAcknowledgment == true when the consumer is either of these subscription types. Another option is exposing the subscription type in the IConsumer interface so that callers could better default the EnsureOrderedAcknowledgement

blankensteiner commented 3 months ago

Hi @smbecker Thanks for the bug report. I like your suggestion of adding the subscription type to IConsumer and then making sure cumulative acknowledgments are not used with the shared or key_shared subscriptions. I'll have a look at that :-)

smbecker commented 2 months ago

I see that this was merged into 3.3.1