search

rbeckman-nextgen / test-mc4

0 stars 0 forks source link

FTP Reader can hang and cannot be stopped or halted #2955

Closed rbeckman-nextgen closed 4 years ago

rbeckman-nextgen commented 4 years ago

A client had a problem where their channel with a FTP reader was hanging and could not be stopped. After obtaining a thread dump, I found this thread

Thread 19259: (state = IN_NATIVE)

We can see that it appears to be blocked when we call the commons.net method FTPClient.listFiles. Looking around online, others have reported similar problems with FTPClient randomly hanging.

http://mail-archives.apache.org/mod_mbox/commons-user/201209.mbox/%3COF7F02830F.CA5C36EA-ONC1257A79.002C855A-C1257A79.00311BE8@kisters.de%3E http://stackoverflow.com/questions/9706968/apache-commons-ftpclient-hanging https://issues.apache.org/jira/browse/NET-61

The problem for us is that if this happens, it requires a server restart. I still haven't been able to reproduce the problem yet, but some have reported that calling setDataTimeout before attempting to connect can solve the problem, or at least get it to throw an exception instead of blocking.

We could also try updating our commons.net library, we are using 3.0.1 while the latest version is 3.3

It could also be a firewall or some other network setting that is causing this problem, as we probably would have run into this problem by now if there wasn't a more specific requirement for it to happen.

Imported Issue. Original Details: Jira Issue Key: MIRTH-3062 Reporter: wayneh Created: 2013-11-14T18:27:55.000-0800

rbeckman-nextgen commented 4 years ago

Hey the client just emailed me saying all of his channels won't deploy now. I've also attached the logs that he sent me.

Imported Comment. Original Details: Author: williaml Created: 2013-11-15T13:18:26.000-0800

rbeckman-nextgen commented 4 years ago

All of them? When you say "won't deploy", you mean that the server started up and literally no channels have been successfully deployed?

I can see errors relating to the Delimited Text batch adaptor, that appear to be due to MIRTH-3002. To fix those errors, edit any channels that use Delimited Text and change the record delimiter to "\n" wherever it's currently blank.

Imported Comment. Original Details: Author: narupley Created: 2013-11-15T13:24:37.000-0800

rbeckman-nextgen commented 4 years ago

Here's what the client did after I e-mailed him Wayne's request to stop all channels and restart MC:

William,

I meant to get back to you…. I had to force a restart of the appliance from the console so I could get an important channel to deploy in MIRTH today. I left the other channel in its original state and it worked also after the restart. However I don’t trust the channel in question since it raises all kinds of errors in my local install of MIRTH (same versioning) BUT it’s a simple mapping, database and file moving channel… no real tricky script in it but the log (that I sent) shows a null exception.

So… What you’ve asked for has been done in the last 30 minutes and all channels have been left alone from their initial deployed state after the restart.

Thanks,

Imported Comment. Original Details: Author: williaml Created: 2013-11-15T15:20:06.000-0800

rbeckman-nextgen commented 4 years ago

He also left this in the email as part of the Local_mirth.log:

java.util.ArrayList cannot be cast to [Ljava.lang.String; com.mirth.connect.plugins.mapper.MapperStepPlugin.buildRegexArray(MapperStepPlugin.java:136) com.mirth.connect.plugins.mapper.MapperStepPlugin.getScript(MapperStepPlugin.java:104) com.mirth.connect.client.ui.editors.transformer.TransformerPane.buildStepList(TransformerPane.java:1045) com.mirth.connect.client.ui.editors.transformer.TransformerPane.saveData(TransformerPane.java:686) com.mirth.connect.client.ui.editors.transformer.TransformerPane.TransformerListSelected(TransformerPane.java:616) com.mirth.connect.client.ui.editors.transformer.TransformerPane.access$400(TransformerPane.java:85) com.mirth.connect.client.ui.editors.transformer.TransformerPane$11.valueChanged(TransformerPane.java:566) javax.swing.DefaultListSelectionModel.fireValueChanged(Unknown Source) javax.swing.DefaultListSelectionModel.fireValueChanged(Unknown Source) javax.swing.DefaultListSelectionModel.setValueIsAdjusting(Unknown Source) javax.swing.plaf.basic.BasicTableUI$Handler.setValueIsAdjusting(Unknown Source) javax.swing.plaf.basic.BasicTableUI$Handler.mouseReleased(Unknown Source) java.awt.AWTEventMulticaster.mouseReleased(Unknown Source) java.awt.AWTEventMulticaster.mouseReleased(Unknown Source) java.awt.AWTEventMulticaster.mouseReleased(Unknown Source) java.awt.Component.processMouseEvent(Unknown Source) javax.swing.JComponent.processMouseEvent(Unknown Source) java.awt.Component.processEvent(Unknown Source) java.awt.Container.processEvent(Unknown Source) java.awt.Component.dispatchEventImpl(Unknown Source) java.awt.Container.dispatchEventImpl(Unknown Source) java.awt.Component.dispatchEvent(Unknown Source) java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source) java.awt.LightweightDispatcher.processMouseEvent(Unknown Source) java.awt.LightweightDispatcher.dispatchEvent(Unknown Source) java.awt.Container.dispatchEventImpl(Unknown Source) java.awt.Window.dispatchEventImpl(Unknown Source) java.awt.Component.dispatchEvent(Unknown Source) java.awt.EventQueue.dispatchEventImpl(Unknown Source) java.awt.EventQueue.access$200(Unknown Source) java.awt.EventQueue$3.run(Unknown Source) java.awt.EventQueue$3.run(Unknown Source) java.security.AccessController.doPrivileged(Native Method) java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) java.awt.EventQueue$4.run(Unknown Source) java.awt.EventQueue$4.run(Unknown Source) java.security.AccessController.doPrivileged(Native Method) java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source) java.awt.EventQueue.dispatchEvent(Unknown Source) java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source) java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source) java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) java.awt.EventDispatchThread.pumpEvents(Unknown Source) java.awt.EventDispatchThread.pumpEvents(Unknown Source) java.awt.EventDispatchThread.run(Unknown Source)

Imported Comment. Original Details: Author: williaml Created: 2013-11-15T15:20:43.000-0800

rbeckman-nextgen commented 4 years ago

I was never able to duplicate this problem in the wild, but I was able to cause it to happen by setting up my own FTP server and blocking a response from being sent. It is very likely that something funny is going on with their FTP server, but regardless there should be ways for Mirth Connect to recover from these instances. We've made two changes which should alleviate these issues.

  1. The timeout field in the File Reader settings now affects the data socket for FTP connections. What this means for this particular client is that if this ever happens again, the connection will timeout and the poll connector will try again based on their poll interval. It should no longer become stuck at polling for longer than the timeout.

  2. However, if the user sets a timeout of 0 to indicate indefinite timeout, it is still possible for the FTP client to be blocked here, so we have also made halt able to break these connections. If the user sets a timeout of 60 seconds and tries to halt it before 60 seconds is up, the channel should halt immediately as well.

Imported Comment. Original Details: Author: wayneh Created: 2013-11-20T17:04:56.000-0800

rbeckman-nextgen commented 4 years ago

Verified that the File Reader in FTP mode can now be halted if it's hanging on a socket read for whatever reason. Also the timeout now actually applies for the data socket as well as the command socket, so just stopping it should work as well, as long as the timeout isn't indefinite.

Imported Comment. Original Details: Author: narupley Created: 2013-12-03T16:20:34.000-0800

rbeckman-nextgen commented 4 years ago

I'm still experiencing these symptoms in 3.0.1 - can't comment on whether or not it is the same root problem, but my SFTP source connector hangs on 'polling'. It ultimately (after timeout or halt, pause/stop does not work) moves the file from the import directory to the designated 'processed' folder, but without renaming it and without processing its contents.

Imported Comment. Original Details: Author: pulverize Created: 2014-03-26T11:28:28.000-0700

rbeckman-nextgen commented 4 years ago

What you are experiencing would be a separate issue, possibly a duplicate of http://www.mirthcorp.com/community/issues/browse/MIRTH-1788, because FTP and SFTP are handled completely differently. Note that this JIRA issue was created because the FTP reader could hang and was not able to be halted. You mentioned that you are able to halt your channel so that also indicates a different problem. If you determine that your issue is not the same as the one I linked and you still think what you're experiencing is a bug, please create a new issue and provide as much detail as possible.

Imported Comment. Original Details: Author: wayneh Created: 2014-03-26T11:41:56.000-0700