Channels under high load can randomly stop working because the Velocity engine hangs

[rbeckman-nextgen]

This appears to be an issue with Velocity: VELOCITY-775.

During load testing for MIRTH-3135, I've noticed it happen twice. Here's what a threaddump shows: \ \ `Thread 15391: (state = BLOCKED)

• org.apache.velocity.runtime.parser.Parser.jj_scan_token(int) @bci=170, line=3580 (Interpreted frame)
• org.apache.velocity.runtime.parser.Parser.jj_3R_60() @bci=3, line=3387 (Interpreted frame)
• org.apache.velocity.runtime.parser.Parser.jj_3R_33() @bci=1, line=2952 (Interpreted frame)
• org.apache.velocity.runtime.parser.Parser.jj_3_10() @bci=1, line=3362 (Interpreted frame)
• org.apache.velocity.runtime.parser.Parser.jj_3_9() @bci=17, line=3431 (Interpreted frame)
• org.apache.velocity.runtime.parser.Parser.jj_2_9(int) @bci=19, line=2717 (Interpreted frame)
• org.apache.velocity.runtime.parser.Parser.Reference() @bci=374, line=1489 (Interpreted frame)
• org.apache.velocity.runtime.parser.Parser.Statement() @bci=62, line=357 (Interpreted frame)
• org.apache.velocity.runtime.parser.Parser.jj_consume_token(int) @bci=147, line=3546 (Compiled frame)
• org.apache.velocity.runtime.parser.Parser.process() @bci=329, line=319 (Compiled frame)
• org.apache.velocity.runtime.parser.Parser.parse(java.io.Reader, java.lang.String) @bci=33, line=117 (Interpreted frame)
• org.apache.velocity.runtime.RuntimeInstance.parse(java.io.Reader, java.lang.String, boolean) @bci=65, line=1226 (Interpreted frame)
• org.apache.velocity.runtime.RuntimeInstance.parse(java.io.Reader, java.lang.String) @bci=4, line=1181 (Interpreted frame)
• org.apache.velocity.runtime.RuntimeInstance.evaluate(org.apache.velocity.context.Context, java.io.Writer, java.lang.String, java.io.Reader) @bci=21, line=1297 (Interpreted frame)
• org.apache.velocity.runtime.RuntimeInstance.evaluate(org.apache.velocity.context.Context, java.io.Writer, java.lang.String, java.lang.String) @bci=13, line=1265 (Interpreted frame)
• org.apache.velocity.app.Velocity.evaluate(org.apache.velocity.context.Context, java.io.Writer, java.lang.String, java.lang.String) @bci=7, line=180 (Interpreted frame)
• com.mirth.connect.util.ValueReplacer.evaluate(org.apache.velocity.VelocityContext, java.lang.String) @bci=16, line=211 (Interpreted frame)
• com.mirth.connect.util.ValueReplacer.replaceValues(java.lang.String, com.mirth.connect.donkey.model.message.ConnectorMessage) @bci=21, line=142 (Interpreted frame)
• com.mirth.connect.connectors.vm.VmDispatcher.replaceConnectorProperties(com.mirth.connect.donkey.model.channel.ConnectorProperties, com.mirth.connect.donkey.model.message.ConnectorMessage) @bci=31, line=81 (Interpreted frame)
• com.mirth.connect.donkey.server.channel.DestinationConnector.process(com.mirth.connect.donkey.server.data.DonkeyDao, com.mirth.connect.donkey.model.message.ConnectorMessage, com.mirth.connect.donkey.model.message.Status) @bci=65, line=305 (Interpreted frame)
• com.mirth.connect.donkey.server.channel.DestinationChain.call() @bci=461, line=224 (Compiled frame)
• com.mirth.connect.donkey.server.channel.Channel.process(com.mirth.connect.donkey.model.message.ConnectorMessage, boolean) @bci=1306, line=1422 (Compiled frame)
• com.mirth.connect.donkey.server.channel.Channel.dispatchRawMessage(com.mirth.connect.donkey.model.message.RawMessage) @bci=189, line=955 (Interpreted frame)
• com.mirth.connect.donkey.server.channel.SourceConnector.dispatchRawMessage(com.mirth.connect.donkey.model.message.RawMessage, boolean) @bci=63, line=177 (Interpreted frame)
• com.mirth.connect.donkey.server.channel.SourceConnector.dispatchRawMessage(com.mirth.connect.donkey.model.message.RawMessage) @bci=3, line=154 (Compiled frame)
• com.mirth.connect.connectors.jdbc.DatabaseReceiver.processRecord(java.util.Map) @bci=15, line=207 (Compiled frame)
• com.mirth.connect.connectors.jdbc.DatabaseReceiver.processResultSet(java.sql.ResultSet) @bci=90, line=173 (Compiled frame)
• com.mirth.connect.connectors.jdbc.DatabaseReceiver.poll() @bci=253, line=122 (Interpreted frame)
• com.mirth.connect.donkey.server.channel.PollConnector$PollConnectorTask.run() @bci=28, line=141 (Interpreted frame)
• java.util.TimerThread.mainLoop() @bci=221, line=555 (Interpreted frame)
• java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame) `

In this case it happened with the Database Reader channel I was testing with. The main processing thread is the one that hung, while replacing the destination connector properties, before placing the message in the destination queue. Interestingly, stopping the channel worked fine. That stop operation shouldn't have interrupted the main processing thread, but apparently it did something to jar Velocity out of the rut it was stuck in.

Imported Issue. Original Details: Jira Issue Key: MIRTH-3136 Reporter: narupley Created: 2014-02-02T07:46:37.000-0800

[rbeckman-nextgen]

Verify if this issue occurs in 2.x branch

Imported Comment. Original Details: Author: wayneh Created: 2014-03-03T11:41:22.000-0800

[rbeckman-nextgen]

I've only been able to reproduce this while debugging with Eclipse, so it sounds like the same issue as VELOCITY-775. Also, it happens in 2.x as well. Though it's worse in 2.x because there certain threads seem to block indefinitely, whereas in 3.x it seems to always finish parsing, albeit slowly.

It's still an unresolved issue on the Velocity side, so (other than fixing their code ourselves) it would require upgrading the library, and seeing their revision history it's unlikely that it'd be fixed anytime soon.

Imported Comment. Original Details: Author: narupley Created: 2014-04-09T09:40:35.000-0700