ActiveMQ messages for a group are not getting consumed and keeps piling

[officialgituser]

Affects:
spring-jms v5.3.34 activemq server v5.16.7

Some times messages to a group are not getting dispatched to the consumer. Once this happens to a group then all the new messages to this group are never dispatched. Our prefetch policy is jms.prefetchPolicy.all=0, so I assume the consumer is at fault here for not polling and getting the message rather than broker end.

<bean id="myQueueMessageListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
        <property name="connectionFactory" ref="naJmsConnectionFactory"/>
        <property name="destination" ref="myQueue"/>
        <property name="messageListener" ref="myQueueMessageListener"/>
        <property name="errorHandler" ref="messageErrorHandler"/>
        <property name="concurrentConsumers" value="5"/>
        <property name="maxConcurrentConsumers" value="5"/>
        <property name="cacheLevelName" value="CACHE_NONE"/>
</bean>

Queue Attributes

AverageMessageSize = 1094
UseCache = true
DuplicateFromStoreCount = 0
MemoryPercentUsage = 2
SendDuplicateFromStoreToDLQ = true
Name = MyQueue
BlockedProducerWarningInterval = 30000
MaxMessageSize = 1098
MemoryUsagePortion = 0.0
CacheEnabled = true
destinationType = Queue
BlockedSends = 0
StoreMessageSize = 21426
EnqueueCount = 129503
TotalBlockedTime = 0
Options =
MinMessageSize = 1094
Paused = false
Subscriptions = [org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-6-36135-1716810416395-0_83,consumerId=ID_host2.xyz.org-36135-1716810416395-1_405_21_989, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-1-43479-1716810316394-0_84,consumerId=ID_host1.xyz.org-43479-1716810316394-1_434_4_99058, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-2-44205-1716810327762-0_84,consumerId=ID_host1.xyz.org-44205-1716810327762-1_418_9_102465, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-4-39227-1716810357236-0_82,consumerId=ID_host1.xyz.org-39227-1716810357236-1_411_15_68467, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-5-42325-1716810401292-0_85,consumerId=ID_host2.xyz.org-42325-1716810401292-1_410_26_14445, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-5-42325-1716810401292-0_86,consumerId=ID_host2.xyz.org-42325-1716810401292-1_411_20_49709, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-4-39227-1716810357236-0_86,consumerId=ID_host1.xyz.org-39227-1716810357236-1_408_5_112043, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-6-36135-1716810416395-0_88,consumerId=ID_host2.xyz.org-36135-1716810416395-1_406_32_2173, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-6-36135-1716810416395-0_84,consumerId=ID_host2.xyz.org-36135-1716810416395-1_407_23_37763, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-8-39359-1716810445710-0_85,consumerId=ID_host2.xyz.org-39359-1716810445710-1_413_8_76665, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-2-44205-1716810327762-0_88,consumerId=ID_host1.xyz.org-44205-1716810327762-1_421_12_89436, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-3-37677-1716810342592-0_86,consumerId=ID_host1.xyz.org-37677-1716810342592-1_413_36_397, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-5-42325-1716810401292-0_85,consumerId=ID_host2.xyz.org-42325-1716810401292-1_410_28_16134, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-3-37677-1716810342592-0_82,consumerId=ID_host1.xyz.org-37677-1716810342592-1_412_23_29131, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-6-36135-1716810416395-0_85,consumerId=ID_host2.xyz.org-36135-1716810416395-1_408_21_36185, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-1-43479-1716810316394-0_85,consumerId=ID_host1.xyz.org-43479-1716810316394-1_435_20_39686, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-7-37591-1716810429987-0_83,consumerId=ID_host2.xyz.org-37591-1716810429987-1_408_4_107929, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-1-43479-1716810316394-0_86,consumerId=ID_host1.xyz.org-43479-1716810316394-1_437_28_22665, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-8-39359-1716810445710-0_82,consumerId=ID_host2.xyz.org-39359-1716810445710-1_410_1_97736, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-1-43479-1716810316394-0_88,consumerId=ID_host1.xyz.org-43479-1716810316394-1_432_24_25473, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-4-39227-1716810357236-0_86,consumerId=ID_host1.xyz.org-39227-1716810357236-1_408_1_95508, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-3-37677-1716810342592-0_85,consumerId=ID_host1.xyz.org-37677-1716810342592-1_410_19_54486, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-6-36135-1716810416395-0_84,consumerId=ID_host2.xyz.org-36135-1716810416395-1_407_22_46860, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-4-39227-1716810357236-0_87,consumerId=ID_host1.xyz.org-39227-1716810357236-1_409_10_89816, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-7-37591-1716810429987-0_88,consumerId=ID_host2.xyz.org-37591-1716810429987-1_407_16_6572, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-3-37677-1716810342592-0_84,consumerId=ID_host1.xyz.org-37677-1716810342592-1_417_20_30062, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-2-44205-1716810327762-0_85,consumerId=ID_host1.xyz.org-44205-1716810327762-1_419_23_21442, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-1-43479-1716810316394-0_84,consumerId=ID_host1.xyz.org-43479-1716810316394-1_434_21_17524, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-5-42325-1716810401292-0_87,consumerId=ID_host2.xyz.org-42325-1716810401292-1_415_32_6164, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-5-42325-1716810401292-0_83,consumerId=ID_host2.xyz.org-42325-1716810401292-1_416_31_6489, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-3-37677-1716810342592-0_85,consumerId=ID_host1.xyz.org-37677-1716810342592-1_410_16_79233, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-2-44205-1716810327762-0_86,consumerId=ID_host1.xyz.org-44205-1716810327762-1_414_14_5278, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-7-37591-1716810429987-0_87,consumerId=ID_host2.xyz.org-37591-1716810429987-1_411_13_74596, org.apache.activemq:type=Broker,brokerName=messagebus,destinationType=Queue,destinationName=MyQueue,endpoint=Consumer,clientId=worker-7-37591-1716810429987-0_85,consumerId=ID_host2.xyz.org-37591-1716810429987-1_404_12_248]
PrioritizedMessages = false
DLQ = false
CursorPercentUsage = 0
MessageGroupType = cached
TempUsageLimit = 268435456
CursorMemoryUsage = 348644
AverageEnqueueTime = 461648.1064275176
AlwaysRetroactive = false
ForwardCount = 0
DispatchCount = 129459
AverageBlockedTime = 0.0
destinationName = MyQueue
MemoryLimit = 2097152
MaxProducersToAudit = 64
ProducerFlowControl = true
QueueSize = 44
ConsumerCount = 34
MaxEnqueueTime = 1014203
MessageGroups = {mygroupId03=ID:host2.xyz.org-37591-1716810429987-1:404:12:248}
TempUsagePercentUsage = 0
CursorFull = false
MaxAuditDepth = 10000
MemoryUsageByteCount = 48136
InFlightCount = 0
ExpiredCount = 0
brokerName = messagebus
DequeueCount = 129459
MaxPageSize = 200
MinEnqueueTime = 3343
ProducerCount = 0
type = Broker

From the EnqueueCount and DispatchCount, it's clear that there are 44 messages that were not dispatched to the consumer. All these 44 messages belong to same group(JMXGroupId - mygroupId03) and the consumer for this group (host2.xyz.org-37591-1716810429987-1_404_12_248) that belongs to java process (worker-7-37591) consumes messages from queue without issue. Its just that every new message with JMXGroupId - mygroupId03 keeps piling up without ever getting dispatched(or polled by the consumer).

jstack.txt

[snicoll]

From the report, it's unclear why you think this is a problem with DefaultMessageListenerContainer. To confirm that, can you validate this does not happen when you use the JMS API directly? Or, alternatively, share a small sample we can run ourselves that reproduces the behavior you've described.

[officialgituser]

Thanks for getting back.. We finally got a clue that those listener threads were stuck on a socket.read awaiting response from DB and since we dont have any sockettimeouts configured, the wait was indefinite.