Problem with connection closure on MQ queues

[jkoguciuk]

mq-jms-spring-boot-starter:3.0.0 Java17

Description of the problem Our company has found a problem related to termination of connection to MQ queues. The problem appears when we try to connect to existing queue and there are temporary problems with network infrastructure and application can`t connect to queue manager. We can also reproduce that problem during connection to non-existent queue.

We try to attach JMS Listener to MQ queue using @JmsListener annotation in order to receive messages. The library throws an Exception that queue is not found and closes the connection. Then application try once again to establish a connection to the queue. The situation is repeated many times. Majority of the connections is successfully closed but some of them not. In our opinion this is a few percent of connections that are not closed. The big problem for us is when connection pool to MQServer is exhausted because the only way to correct application operation is to restart it.

Code example of JmsListener

@Component
public class QueueMessageListener {
    static final String TESTQUEUE2 = "TESTQ2";
    private final static Logger LOGGER =
            Logger.getLogger(Logger.GLOBAL_LOGGER_NAME);

    @JmsListener(id = "listener2", destination = TESTQUEUE2, concurrency = "10-20")
    void receive(String textMessage) throws Exception {
        LOGGER.log(Level.INFO, "Queue: " + TESTQUEUE2 + " received
    }
}

In order to check the quantity of connections established from our application to queue manager we prepared batch file:

@echo off
SET  MQSERVER=CHANNEL/TCP/SERVER_IP(PORT) 
echo MQ connections count: 
:here
echo DISPLAY CHSTATUS(CHANNEL) CONNAME | runmqsc QUEUEMANAGER | find /c "CONNAME(IP_ADDRESS)"
goto here

CHANNEL, SERVER_IP, PORT, CHANNEL, QUEUEMANAGER are replaced by real values of course.

Testing the problem We have test application using JmsListener as above. TESTQUEUE2 is the queue which does not exist on MQ SERVER. The number of connections shown by batch file should be constant but after some minutes it grows slowly. In about one hour number of connections exceeds limit in queue manager and application terminates because of inability of queue connection creation.

[ibmmqmet]

I don't think this is anything to do with this particular package as it doesn't get involved in destroying objects.

When I ran a test, I did confirm the problem of apparent connection leakage. However it was only if I set any lower-bound in the concurrency attribute. If I just had "concurrency=10", then the application behaved as expected - there was still an exception and a retried connection, but the actual number stayed constantly low.

So I think this is something odd in the CachingConnectionFactory implementation. And therefore something to be raised in the base Spring Framework repo. They might be able to suggest additional diagnostics as well.

[jkoguciuk]

Thank you for your reply. I have tested it on our test application with concurrency="10" and concurrency="50" parameters. In both cases it turned out that number of not closed connections decreased by ~70%. When we had concurrency = "10-20" number of not closed connections was about 20 per hour and now we have about 6-7 per hour. Unfortunately there is still growth.

[Swierkowski]

I don't think this is anything to do with this particular package as it doesn't get involved in destroying objects.

When I ran a test, I did confirm the problem of apparent connection leakage. However it was only if I set any lower-bound in the concurrency attribute. If I just had "concurrency=10", then the application behaved as expected - there was still an exception and a retried connection, but the actual number stayed constantly low.

So I think this is something odd in the CachingConnectionFactory implementation. And therefore something to be raised in the base Spring Framework repo. They might be able to suggest additional diagnostics as well.

According to javadoc when you set concurrency "10" it is actually 1-10. So, there is always a range. It could be that setting it implicitly to 1-10 makes the issue less visible.

https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/jms/annotation/JmsListener.html

Are you sure that it is related to CachingConnectionFactory? It is observed only with IBM MQ provider. Could it be that closing the connection is just too slow for IBM MQ?

[ibmmqmet]

Again, I can see no way that this particular module is involved here as it only deals with creation of objects, not destruction.

Looking at the CachingConnection implementation would be my first suggestion as this is clearly something to do with creating and not successfully closing objects. Whether that's because of a problem in the caching component, or in the MQ JMS client code, I can't tell. If the latter, then that would have to be dealt with via an IBM support case. One thing I noticed is that switching from the spring.cache to the pooled connection support did not appear to result in uncontrolled growth in connections.

One thing that is important to realise, is that both JMS Connections and Sessions result in an MQCONN showing up in the queue manager. So there might be, for example, dangling sessions lying around when a Connection is closed. And that might be part of the problem. Although I'd hope/expect sessions are then cleaned up, that might be one area for investigation.

[Jenson3210]

@jkoguciuk did you manage to find the root cause here? We seem to be facing similar issues.