Cache level CACHE_NONE does not recover properly on JMSException [SPR-11762]

[spring-projects-issues]

liangwenbo opened SPR-11762 and commented

Hi： I use spring-jms consume activemq5.9 queue, and it stop consuming when messages class cannot found in consumer's classloader, and when messages in two queues cannot found class both, which is easier to reproduce. Here's my configure:

\ \ \

<jms:listener-container connection-factory="cachingConnectionFactory" concurrency="10">
    <jms:listener destination="testq" ref="messageReceiver" method="receiveMessage"/>
    <jms:listener destination="testl" ref="messageReceiver2" method="receiveMessage"/>
</jms:listener-container>
<bean id="messageReceiver" class="activemq.MessageReceiver">
</bean>
<bean id="messageReceiver2" class="activemq.MessageReceiver2">
</bean>

---

Affects: 3.1.1

Reference URL: http://forum.spring.io/forum/other-spring-related/remoting/49984-jms-defaultmessagelistenercontainer-recovery-fails

Attachments:

• ClassNotFoundTest.java (932 bytes)
• sprint-jms.xml (1.30 kB)

Issue Links:

• 16400 MessageListenerAdapter might throw JMSException on message deserialization

[spring-projects-issues]

Stéphane Nicoll commented

Can you post the stacktrace of these exceptions please? I am not sure I understand what you are referring to. Better, if you could submit a project that reproduces the issue on the repro repo, that would be great.

[spring-projects-issues]

liangwenbo commented

Here's the test code and configuration

[spring-projects-issues]

liangwenbo commented

Sorry for did not upload the last test code. First, insert some class instance not exist(eg. TestMee) into 'testl' and 'testq' queues; Then, run and stop ClassNotFoundTest for some times, it's easy to found one of the consumers or all is stuck spending messages. Exception info: 2014-05-07 12:59:59 org.springframework.jms.listener.DefaultMessageListenerContainer handleListenerSetupFailure WARN: Setup of JMS message listener invoker failed for destination 'testl' - trying to recover. Cause: Failed to build body from content. Serializable class not available to broker. Reason: java.lang.ClassNotFoundException: TestMee 2014-05-07 12:59:59 下午 org.springframework.jms.listener.DefaultMessageListenerContainer refreshConnectionUntilSuccessful INFO: Successfully refreshed JMS Connection 2014-05-07 12:59:59 下午 org.springframework.jms.listener.AbstractMessageListenerContainer invokeErrorHandler WARN: Execution of JMS message listener failed, and no ErrorHandler has been set. javax.jms.JMSException: Failed to build body from content. Serializable class not available to broker. Reason: java.lang.ClassNotFoundException: TestMee at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:36) at org.apache.activemq.command.ActiveMQObjectMessage.getObject(ActiveMQObjectMessage.java:193) at org.springframework.jms.support.converter.SimpleMessageConverter.extractSerializableFromMessage(SimpleMessageConverter.java:215) at org.springframework.jms.support.converter.SimpleMessageConverter.fromMessage(SimpleMessageConverter.java:103) at org.springframework.jms.listener.adapter.MessageListenerAdapter.extractMessage(MessageListenerAdapter.java:407) at org.springframework.jms.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:345) at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:537) at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:497) at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:468) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:326) at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:264) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1071) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1063) at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:960) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.ClassNotFoundException: TestMee at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.activemq.util.ClassLoadingAwareObjectInputStream.load(ClassLoadingAwareObjectInputStream.java:87) at org.apache.activemq.util.ClassLoadingAwareObjectInputStream.resolveClass(ClassLoadingAwareObjectInputStream.java:46) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.activemq.command.ActiveMQObjectMessage.getObject(ActiveMQObjectMessage.java:191) ... 13 more

[spring-projects-issues]

Stéphane Nicoll commented

I don't see anything obviously wrong here. There is an exception, the connection is refreshed successfully and the message is processed again but it fails again for the same reason. Are you saying that after some time, these exceptions goes away and if you send additional messages to that queue, they are not processed by the listener?

[spring-projects-issues]

Stéphane Nicoll commented

I have created a repro project based on your simple example and I can't reproduce the issue. See https://github.com/spring-projects/spring-framework-issues/tree/master/SPR-11762

The exception that you mention is indeed there (as it should) but if I send additional messages, these are handled (and they failed again for the same reason). I simply put the class in src/test/java and the consumer in src/main/java. The classpath of the application does not take test resources so I can have both running at the same time.

And everything is working as expected here. Can you please try it out and let me know what I am missing? Thanks!

[spring-projects-issues]

liangwenbo commented

Thanks for your attentions, I've sent a pull request on https://github.com/spring-projects/spring-framework-issues/tree/master/SPR-11762. it midofied the spr-11762 test code with jms:listener-container and CachingConnectionFactory config, and sent 100 classnotfound messages to 2 queue, then it was reproduce when running ReproApp: some of the two consumer are existing but stuck spending, left the rest of messages in the queues.

[spring-projects-issues]

Stéphane Nicoll commented

Thank you! This is interesting because I used the new @JmsListener infrastructure that is available as from the upcoming 4.1 version as a showcase but switching to the existing infrastructure reveals indeed the problem. It's funny because it's mostly syntactic sugar on top of the existing infrastructure so I am really surprised that it's failing here and not with my use case.

I can reproduce the issue now, I'll look into it. Thanks again.

[spring-projects-issues]

Stéphane Nicoll commented

The reason why this works with the new infrastructure is that the handling of the serialization is slightly different and throw a different exception type which does not lead to a recovery.

I traced down the problem to the fact that you are using the default cache with no transaction manager. When this happens CACHE_NONE is used (i.e. no caching of resources whatsoever).

If you change the cache level to CACHE_CONNECTION, the recovery works as expected (add cache="connection" to your jms:listener-container element).

I am adding this to the backlog to further investigate why recovery is broken when the resources are not cached. We might also update the existing code to have the same behaviour as the new infrastructure. That way, a JMSException would not be thrown and would not lead to a recovery. I'll give it some thoughts.

[spring-projects-issues]

liangwenbo commented

Hi Stephane, I use the cache="connection" param run the test case, the consumers still encountered stuck sometimes, although the probability was much smaller. And, it seems DefaultMessageListenerContainer's recovery mechanism was clashed with CachingConnectionFactory, if DefaultMessageListenerContainer's connectionFactory not a CachingConnectionFactory bean, the test case will be passed.

[spring-projects-issues]

Stéphane Nicoll commented

Interesting, thanks.

[spring-projects-issues]

Stéphane Nicoll commented

Hi there, as we discussed already, I have created #16400 to avoid throwing a JMSException in such a case. It will not fix this actual issue but at least your scenario won't lead to an invalidation of the connection (i.e. no recovery at all).

[spring-projects-issues]

liangwenbo commented

Thanks Stephane‘s effort.