search

ibm-messaging / mq-jms-spring

Components to assist MQ JMS integration with Spring frameworks
Apache License 2.0
189 stars 102 forks source link

Consumer stopped working after exception trying rollback() when putting session back into the pool, will invalidate #99

Closed monnetchr closed 5 months ago

monnetchr commented 1 year ago

Please include the following information in your ticket.

Spring Boot config for JMS:

spring:
  jms:
    pub-sub-domain: false
    listener:
      acknowledge-mode: client

The consequence was fatal: the app simply stopped to consume messages.

[2023-08-10 13:07:36,574] [ourJmsListener-1] WARN  org.messaginghub.pooled.jms.JmsPoolSession - Caught exception trying rollback() when putting session back into the pool, will invalidate. com.ibm.msg.client.jms.DetailedJMSException: JMSCMQ0002: The method 'MQBACK' failed.
An IBM MQ call failed.
Please see the linked exception for more information.
com.ibm.msg.client.jms.DetailedJMSException: JMSCMQ0002: The method 'MQBACK' failed.
    at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:595)
    at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
    at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:253)
    at com.ibm.msg.client.wmq.internal.WMQSession.syncpoint(WMQSession.java:1989)
    at com.ibm.msg.client.wmq.internal.WMQSession.rollback(WMQSession.java:1833)
    at com.ibm.msg.client.jms.internal.JmsSessionImpl.rollbackTransaction(JmsSessionImpl.java:2878)
    at com.ibm.msg.client.jms.internal.JmsSessionImpl.rollback(JmsSessionImpl.java:2101)
    at com.ibm.mq.jms.MQSession.rollback(MQSession.java:952)
    at org.messaginghub.pooled.jms.JmsPoolSession.close(JmsPoolSession.java:112)
    at brave.jms.TracingSession.close(TracingSession.java:123)
    at org.springframework.jms.support.JmsUtils.closeSession(JmsUtils.java:109)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.clearResources(DefaultMessageListenerContainer.java:1289)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1135)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
    at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
    ... 12 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009;AMQ9206: Error sending data to host 'OUR_MQ_HOST:1414'. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2009],3=OUR_MQ_HOST:1414,4=TCP,5=RemoteTCPConnection.send(byte [ ],int,int,int,int)]
    at com.ibm.mq.jmqi.remote.impl.RemoteTCPConnection.send(RemoteTCPConnection.java:1824)
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection.wrapSend(RemoteConnection.java:3026)
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection.sendTSH(RemoteConnection.java:2792)
    at com.ibm.mq.jmqi.remote.impl.RemoteSession.sendTSH(RemoteSession.java:793)
    at com.ibm.mq.jmqi.remote.impl.RemoteSession.sendTSH(RemoteSession.java:709)
    at com.ibm.mq.jmqi.remote.api.RemoteFAP.MQBACK(RemoteFAP.java:9978)
    at com.ibm.mq.jmqi.monitoring.JmqiInterceptAdapter.MQBACK(JmqiInterceptAdapter.java:133)
    at com.ibm.msg.client.wmq.internal.WMQSession.syncpoint(WMQSession.java:1968)
    ... 10 common frames omitted
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2009
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection.asyncConnectionBroken(RemoteConnection.java:3890)
    at com.ibm.mq.jmqi.remote.impl.RemoteConnection.notifyReconnect(RemoteConnection.java:4925)
    at com.ibm.mq.jmqi.remote.impl.RemoteRcvThread.run(RemoteRcvThread.java:515)
    at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
    at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
    at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:343)
    at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
    at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1240)
monnetchr commented 1 year ago

The issue also happened with Spring Boot and mq-jms-spring-boot-starter 3.1.2 on openjdk version "17.0.8" 2023-07-18 LTS

ibmmqmet commented 1 year ago

You have given no information about your application or what is really going on in your environment.

But the key line in the stack appears to be: Caused by: com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').

And that suggests a fatal error on the connection to the qmgr. So it should be expected that there is nothing remaining that can be put back into a session/connection pool.

monnetchr commented 1 year ago

Thanks for your answer. There is nothing special in the app, it has a dependency on mq-jms-spring-boot-starter and a bean with @JmsListener annotation. What happened on the environment: one node of the queue manager cluster failed. The client connects to OUR_MQ_HOST:1414 which is the cluster address, the app config knows nothing of the nodes themselves. The exception above was the last thing that the app logged, after that it stopped consuming messages. This is the main issue. The expected behaviour is that the MQ client should failover to another node. Since the exception from the MQ Client is not relevant, how can I investigate the problem further?

saikrishnaradarapu commented 5 months ago

Hi @monnetchr, Did you get any solution?

monnetchr commented 5 months ago

Well, we migrated to Kafka 😄

Joke aside, it was not an issue with mq-jms-spring but with our corporate MQ setup where client connections could timeout and fail, so what can the client library do? Nothing.