JDBC connection cannot be closed on cluster switch back

[rbeckman-nextgen]

1. Create a channel with a database reader on an Oracle database and the default "keep connection open" parameter to true
2. Stop the cluster primary server
3. The cluster secondary server takes over
4. Start the cluster primary server
5. The cluster primary server switches back

At this stage, the cluster secondary server tries to pause the channel. This never happens: the channel is blocked in the "pausing" state.

The issue boils down to the following method never returning: com.mirth.connect.connectors.jdbc.DatabaseReceiverQuery.closeSelectConnection

More specifically, this statement never returns:

Imported Issue. Original Details: Jira Issue Key: MIRTH-3559 Reporter: daniel Created: 2014-12-22T03:11:32.000-0800

[rbeckman-nextgen]

End of description ...

More specifically, this statement never returns: selectConnection.close();

Changing the way the connection is handled solves the problem (only for Oracle of course), e.g.:

    Properties p = new Properties();
    p.setProperty("user", username);
    p.setProperty("password", password);
    p.setProperty(OracleConnection.CONNECTION_PROPERTY_THIN_READ_TIMEOUT, "5000");
    selectConnection = DriverManager.getConnection(url, p);
    selectConnection.setAutoCommit(true);

Instead of:

           selectConnection = DriverManager.getConnection(url, username, password);

Indeed, the code above then throws the following error:

---

[2014-12-22 12:02:40,027] ERROR (com.mirth.connect.connectors.jdbc.DatabaseReceiverQuery:359): Failed to close database connection java.sql.SQLRecoverableException: IO Error: Socket read timed out at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:857) at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2498) at com.mirth.connect.connectors.jdbc.DatabaseReceiverQuery.closeSelectConnection(DatabaseReceiverQuery.java:355) at com.mirth.connect.connectors.jdbc.DatabaseReceiverQuery.stop(DatabaseReceiverQuery.java:127) at com.mirth.connect.connectors.jdbc.DatabaseReceiver.onStop(DatabaseReceiver.java:87) at com.mirth.connect.donkey.server.channel.SourceConnector.stop(SourceConnector.java:121) at com.mirth.connect.donkey.server.channel.PollConnector.stop(PollConnector.java:45) at com.mirth.connect.donkey.server.channel.Channel.pause(Channel.java:779) at com.mirth.connect.server.controllers.DonkeyEngineController$ChannelStatusTask.execute(DonkeyEngineController.java:1361) at com.mirth.connect.server.channel.ChannelTask.call(ChannelTask.java:59) at com.mirth.connect.server.channel.ChannelTask.call(ChannelTask.java:16) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)Caused by: oracle.net.ns.NetException: Socket read timed out at oracle.net.ns.Packet.receive(Packet.java:350) at oracle.net.ns.DataPacket.receive(DataPacket.java:105) at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:305) at oracle.net.ns.NetInputStream.read(NetInputStream.java:249) at oracle.net.ns.NetInputStream.read(NetInputStream.java:171) at oracle.net.ns.NetInputStream.read(NetInputStream.java:89) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123) at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79) at oracle.jdbc.driver.T4CMAREngineStream.unmarshalUB1(T4CMAREngineStream.java:429) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:397) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257) at oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:59) at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:844) ... 14 more

---

The consequence of this issue is that the Mirth Connect server cannot be stopped. The cluster switch is then of course broken too.

An Appliance restart (host reboot) is necessary for Mirth Connect to recover!

A workaround is to set the parameter "Keep connection open" to false whenever deploying on a cluster.

---

The cause of the problem is maybe the way the virtual IP address of the cluster is used by the Oracle driver?

Imported Comment. Original Details: Author: daniel Created: 2014-12-22T03:20:21.000-0800

[rbeckman-nextgen]

Alternative solution: upgrade from jdbc 4.0 to 4.1 (java 7) and use one of the following methods, probably with the same results: http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#abort(java.util.concurrent.Executor) http://docs.oracle.com/javase/7/docs/api/java/sql/Connection.html#setNetworkTimeout(java.util.concurrent.Executor,%20int)

Imported Comment. Original Details: Author: daniel Created: 2014-12-22T05:03:35.000-0800

[rbeckman-nextgen]

Do you think this issue can be fixed before 3.4.0 ?

In our appliances cluster setup, we have to reboot the secondary server each time after a cluster failover happens. (n) Otherwise the secondary server remains stuck with all database reader channels trying to stop.

Also the workaround mentioned before doesn't work for us : this issue happens whatever value the parameter "Keep connection open" is set to.

Imported Comment. Original Details: Author: ludwig Created: 2015-07-09T01:43:21.000-0700

[rbeckman-nextgen]

Another thing that is broken by this issue : As mirth connect is stuck with these DBReader channels trying to stop, it is not possible to update mirth connect using the appliance control panel software updates functionality. Trying to do so there is an error message : {quote} Script 10-stop-mirthconnect.sh exited with error. Please contact support for assistance Installer exited with unexpected status: 1 {quote} (see screenshot attached)

Imported Comment. Original Details: Author: ludwig Created: 2015-07-09T05:40:24.000-0700