"Unexpected end of file from server" exception

[danielkoo]

Hi Chris,

I have been getting this error during random times going back and forth between volunteers; it's been twice so far - each time I am forced to ctrl-C and run the app again as it stays hung:

Here is the error. Thanks Chris.

Jul 19, 2015 10:08:19 PM org.apache.cayenne.remote.BaseConnection sendMessage
INFO: *** Message error for 230: Query - took 8 ms.
Exception in thread "AWT-EventQueue-0" org.apache.cayenne.CayenneRuntimeException: [v.3.0.2 Jun 11 2011 09:52:20] Remote error. URL - http://www.strandz.org/cayenneRemoteService/cayenne-service; CAUSE - Unexpected end of file from server
    at org.apache.cayenne.remote.hessian.HessianConnection.doSendMessage(HessianConnection.java:153)
    at org.apache.cayenne.remote.BaseConnection.sendMessage(BaseConnection.java:72)
    at org.apache.cayenne.remote.ClientChannel.send(ClientChannel.java:323)
    at org.apache.cayenne.remote.ClientChannel.onQuery(ClientChannel.java:113)
    at org.apache.cayenne.util.ObjectContextQueryAction.runQuery(ObjectContextQueryAction.java:334)
    at org.apache.cayenne.util.ObjectContextQueryAction.executePostCache(ObjectContextQueryAction.java:104)
    at org.apache.cayenne.util.ObjectContextQueryAction.execute(ObjectContextQueryAction.java:91)
    at org.apache.cayenne.CayenneContext.onQuery(CayenneContext.java:384)
    at org.apache.cayenne.CayenneContext.performQuery(CayenneContext.java:374)
    at org.apache.cayenne.util.RelationshipFault.resolveFromDB(RelationshipFault.java:89)
    at org.apache.cayenne.util.PersistentObjectHolder.resolve(PersistentObjectHolder.java:150)
    at org.apache.cayenne.util.PersistentObjectHolder.getValue(PersistentObjectHolder.java:69)
    at org.apache.cayenne.reflect.valueholder.ValueHolderProperty.readProperty(ValueHolderProperty.java:71)
    at org.apache.cayenne.BaseContext.prepareForAccess(BaseContext.java:241)
    at org.strandz.data.wombatrescue.objects.cayenne.client.auto._RosterSlot.getActualDayInWeek(_RosterSlot.java:203)
    at org.strandz.data.wombatrescue.objects.cayenne.client.RosterSlot.getDayInWeek(RosterSlot.java:131)
    at org.strandz.data.wombatrescue.objects.RosterSlotHelper.helperToString(RosterSlotHelper.java:366)
    at org.strandz.data.wombatrescue.objects.cayenne.client.RosterSlot.toString(RosterSlot.java:222)
    at java.lang.String.valueOf(String.java:2847)
    at java.lang.StringBuilder.append(StringBuilder.java:128)
    at java.util.AbstractCollection.toString(AbstractCollection.java:458)
    at org.apache.cayenne.util.PersistentObjectList.toString(PersistentObjectList.java:434)
    at java.lang.String.valueOf(String.java:2847)
    at java.lang.StringBuilder.append(StringBuilder.java:128)
    at org.strandz.lgpl.extent.MethodTie.getFieldValue(MethodTie.java:352)
    at org.strandz.lgpl.extent.ListExtent.setList(ListExtent.java:97)
    at org.strandz.lgpl.extent.AggregationExtent.setList(AggregationExtent.java:72)
    at org.strandz.lgpl.extent.Ties.setNewListVector(Ties.java:172)
    at org.strandz.core.prod.Block.setDataRecords(Block.java:1307)
    at org.strandz.core.prod.CompositeBlock.setDataRecords(CompositeBlock.java:180)
    at org.strandz.core.prod.Block.internalSyncDetail(Block.java:960)
    at org.strandz.core.prod.Block.syncDetail(Block.java:816)
    at org.strandz.core.prod.Block.syncDetail(Block.java:777)
    at org.strandz.core.prod.NavigatingState.next(NavigatingState.java:149)
    at org.strandz.core.prod.EditingState.next(EditingState.java:36)
    at org.strandz.core.prod.OperationsProcessor.next(OperationsProcessor.java:334)
    at org.strandz.core.interf.LoadInsertDoneState.next(LoadInsertDoneState.java:142)
    at org.strandz.core.interf.Strand.cap(Strand.java:2388)
    at org.strandz.core.interf.Strand.controlActionPerformed(Strand.java:2457)
    at org.strandz.core.interf.Strand.execute(Strand.java:2061)
    at org.strandz.core.interf.NodeController.execute(NodeController.java:385)
    at org.strandz.core.applichousing.PictToolBarController$AllActionListener.actionPerformed(PictToolBarController.java:470)
    at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2018)
    at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2341)
    at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
    at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
    at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
    at java.awt.Component.processMouseEvent(Component.java:6516)
    at javax.swing.JComponent.processMouseEvent(JComponent.java:3312)
    at java.awt.Component.processEvent(Component.java:6281)
    at java.awt.Container.processEvent(Container.java:2229)
    at java.awt.Component.dispatchEventImpl(Component.java:4872)
    at java.awt.Container.dispatchEventImpl(Container.java:2287)
    at java.awt.Component.dispatchEvent(Component.java:4698)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
    at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
    at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
    at java.awt.Container.dispatchEventImpl(Container.java:2273)
    at java.awt.Window.dispatchEventImpl(Window.java:2719)
    at java.awt.Component.dispatchEvent(Component.java:4698)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:747)
    at java.awt.EventQueue.access$300(EventQueue.java:103)
    at java.awt.EventQueue$3.run(EventQueue.java:706)
    at java.awt.EventQueue$3.run(EventQueue.java:704)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
    at java.awt.EventQueue$4.run(EventQueue.java:720)
    at java.awt.EventQueue$4.run(EventQueue.java:718)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:717)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
Caused by: java.net.SocketException: Unexpected end of file from server
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at com.caucho.hessian.client.HessianProxy.invoke(HessianProxy.java:168)
    at com.sun.proxy.$Proxy4.processMessage(Unknown Source)
    at org.apache.cayenne.remote.hessian.HessianConnection.doSendMessage(HessianConnection.java:145)
    ... 78 more

[chrismurrph]

No actual problems on the Server for this one, which goes along with you saying that you can continue to work after restarting the Client. We haven't seen these types of errors before. Is it reproducible? If not, do you think a poor Internet connection is a contributing factor?

[danielkoo]

Not sure, but next time it happens I'll test for packet loss on my side - would that help?

On 19 July 2015 at 23:05, Chris Murphy notifications@github.com wrote:

No actual problems on the Server for this one, which goes along with you saying that you can continue to work after restarting the Client. We haven't seen these types of errors before. Is it reproducible? If not, do you think a poor Internet connection is a contributing factor?

— Reply to this email directly or view it on GitHub https://github.com/chrismurrph/strandz/issues/5#issuecomment-122660119.

[chrismurrph]

Not sure. If it happens regularly, and given it hasn't happened before, then it would seem to be to do with this particular Server (JVM instance) rather than a function of the Internet in general. I'm thinking a contributing factor might be the Server (as in web server <=> JVM instance) trying to do too much. There was another webapp running as well - there are three altogether (I only took one out). I'm going to try and change some JVM settings now...

[danielkoo]

Thanks Chris. I'll continue on using it tonight, will let you know if I run into the same thing again. Ta

On Sunday, 19 July 2015, Chris Murphy notifications@github.com wrote:

Not sure. If it happens regularly, and given it hasn't happened before, then it would seem to be to do with this particular Server (JVM instance) rather than a function of the Internet in general. I'm thinking it is caused by the Server (as in web server <=> JVM instance) trying to do too much. There was another webapp running as well - there are three altogether (I only took one out). I'm going to try and change some JVM settings now...

— Reply to this email directly or view it on GitHub https://github.com/chrismurrph/strandz/issues/5#issuecomment-122660787.

[chrismurrph]

Okay I'll be here. All three webapps are running at the moment. This is all I have changed for the JVM:

JAVA_OPTIONS="-XX:MaxPermSize=128m -Xmx1600m"

Compared to before that is more memory and more PermGenSpace. It makes sense that these kinds of things will need to be increased because there is a lot going on, utilizing multiple threads (actors are being used in the other two webapps).
If you get issues tonight I'll be able to take the other two apps offline pretty quickly.

[danielkoo]

It's been super responsive tonight; I was able to add 30-odd vols tonight, which in the past would have been a nightmare wait, but spent very little time waiting for record saves this time. Keep it up :)

On 20 July 2015 at 17:16, Chris Murphy notifications@github.com wrote:

Okay I'll be here. All 3 webapps are running at the moment. This is all I have changed for the JVM:

JAVA_OPTIONS="-XX:MaxPermSize=128m -Xmx1600m"

Compared to before that is more memory and more PermGenSpace. It makes sense that these kinds of things will need to be increased because there is a lot going on, utilizing multiple threads (actors are being used in the other two webapps).

If you get issues tonight I'll be able to take the other two apps offline pretty quickly.

— Reply to this email directly or view it on GitHub https://github.com/chrismurrph/strandz/issues/5#issuecomment-122783820.

[chrismurrph]

Great. The improvement must be due to there being more memory in the JVM. I think we are now at the upper limit for how much Jetty can grab. Good to hear that the rosters job is now a little easier. The application is dealing with such tiny amounts of data it should always be practically instant to do anything. I'll close all the issues I think this simple change has fixed, they can always be reopened...

[danielkoo]

I got another one of these tonight... I'd been doing a number of things so unsure of what I could do to replicate it.. will keep an eye out next time, thanks Chris!

[chrismurrph]

Makes sense now that this particular Client-only problem was not fixed by increasing memory on the Server. On the other hand! - maybe something on the Server is cutting your Cayenne query off. Is this problem less frequent than before? If so I'll squeeze out the last amount of memory available on this machine and we can see if they become even less frequent.

[danielkoo]

Hi Chris,

It's hard to tell atm, as we haven't been running the current config for long enough for me to get an idea of how it behaves, so I cannot tell you whether it's more/less frequent.. I am happy for no changes to be made, and see how I go over the next few roster drafts?

On 22 July 2015 at 22:40, Chris Murphy notifications@github.com wrote:

Makes sense now that this particular Client-only problem was not fixed by increasing memory on the Server. On the other hand! - maybe something on the Server is cutting your Cayenne query off. Is this problem less frequent than before? If so I'll squeeze out the last amount of memory available on this machine and we can see if they become even less frequent.

— Reply to this email directly or view it on GitHub https://github.com/chrismurrph/strandz/issues/5#issuecomment-123707831.

[chrismurrph]

Yes good idea and good to get an awareness as you say, of whether the most recent memory increase made a difference. Once you can say it probably has made a difference then we can tweak the last little bit out of it and see if that makes it even better. All the stack traces say 'Cayenne'. I've left Cayenne at version 3 because there was thinking to do to move onto 3.1. So that's another change that can be tried out once we've explored the whole memory thing fully.

[danielkoo]

Hi Chris,

I just encountered it.. not sure what you can see now on the server itself as it may be transient..

[danielkoo]

just FYI, closing the app and running it again, I am seeing the error soon after logging in, so will restart the app.

[chrismurrph]

I watched a build tools / SBT video recently where they said how when the JVM is doing too much it just stops responding for a long time, which could be what is going on here. I can always make the Linode bigger/faster. For now I will just put the memory up significantly to:

JAVA_OPTIONS="-XX:MaxPermSize=128m -Xmx3000m"

[chrismurrph]

By the way I looked all the way through the logs and there's nothing special about them, no crash or anything. The biggest problem is that logging is set to INFO - so there are way too many messages.

[danielkoo]

if my understanding is correct, only this is running on the VPS, right? if the previous amount of memory was not enough, something is very wrong.. is it leaking perhaps? If so, it's only a matter of time before we get the same error again...

On 30 July 2015 at 05:29, Chris Murphy notifications@github.com wrote:

I watched a build tools / SBT video recently where they said how when the JVM is doing too much it just stops responding for a long time, which could be what is going on here. I can always make the Linode bigger/faster. For now I will just put the memory up significantly to:

JAVA_OPTIONS="-XX:MaxPermSize=128m -Xmx3000m"

— Reply to this email directly or view it on GitHub https://github.com/chrismurrph/strandz/issues/5#issuecomment-126068175.

[chrismurrph]

There are three webapps running that in turn access other programs/actors, all doing work. For instance there is a virtual PLC running all the time. See the UI for some of them here. When you do the rostering lots of new classes are created on the server JVM, that all need to be garbage collected. So far we have not had a OutOfMemoryException. We would need one of them for there to be a leak. Increasing the memory used is something I want to do anyway. The thing that stops a JVM is garbage collection. Increasing the memory will make life easier for the garbage collector.

[danielkoo]

I see... Btw have you looked into any funding options for non profits, so as to separate these related costs from of your account?

On Thursday, 30 July 2015, Chris Murphy notifications@github.com wrote:

There are three webapps running, doing quite a lot of work. For instance there is a virtual PLC running all the time and, well - many other things. See the UI for some of them here http://www.seasoft.com.au/atmosphere. When you do the rostering lots of new classes are created on the server JVM, that all need to be garbage collected. So far we have not had a OutOfMemoryException. We would need one of them for there to be a leak. Increasing the memory used is something I want to do anyway. The thing that stops a JVM is garbage collection. Increasing the memory will make life easier for the garbage collector.

— Reply to this email directly or view it on GitHub https://github.com/chrismurrph/strandz/issues/5#issuecomment-126190601.

[danielkoo]

hey Chris, after using it for a while tonight I am starting to get consistent "Unexpected end of file from server" errors whenever I try to bring up the rosterable workers after logging in... would it require a restart just to get it going for now? I'll have some time to work on Strandz with you this break, so hopefully we can spend time debugging some of the common issues that crop up. Thanks!

[chrismurrph]

As much as anything the problem is to do with too much going on on the server machine. Nothing really wrong, nothing to be fixed. There is no exception on the server. There is no memory leak. The JVM may well be stopping for garbage collection for like 15 seconds. The EOF error seems like a timeout in disguise.

To make sure of this I just brought up Rosterable Workers no problem - without having done anything on the server-machine, except check that there are no errors in the jetty log.

We could always host it elsewhere, where the server-machine is not being so hammered. Could we do something like put it in a Docker container on an AWS machine? What would you be interested in doing? By the way I am now coming back on New Year's Day, so a little later than I thought before.

[danielkoo]

The thing was I was not able to connect to it from that point onwards (I tried a number of times), did you restart anything, or did it just recover by itself? Even if it recovers after some time it would be nice to be able to restart it myself as a quick workaround... what do you think?

Yes, I think we can look into hosting it elsewhere, to be honest, all it ends up doing is just posting the pages to the web server, so it only needs to run when I am rostering, and can be turned off for the rest of the month (as long as your web server is around to serve the roster it's all good)

I am free Tues next week if you are free too? Oh, when do you come back to Sydney first of all? :)

On 23 December 2015 at 05:08, Chris Murphy notifications@github.com wrote:

The problem is to do with too much going on on the server machine. Nothing really wrong, nothing to be fixed. The JVM will be stopping for garbage collection for like 15 seconds. The EOF error is probably just a timeout in disguise. We could always host it elsewhere, where the server-machine is not being so hammered. Could we do something like put it in a Docker container on an AWS machine? What would you be interested in doing?

— Reply to this email directly or view it on GitHub https://github.com/chrismurrph/strandz/issues/5#issuecomment-166692650.

[chrismurrph]

There was another process (tube bundle system) that was emitting warnings to the log that I stopped from doing that. So possibly I eased its workload which fixed things up - many Servers all on the same JVM - which is stretched.You are right it only needs to be run at certain times - but it will be naturally idle when it is not being used - so there's no gain from turning it off and on. And max load is really all that matters b/c I don't want the other Servers to be turned off. Actually anyone could have done what I did to ease the load as it was the UI for another process that I changed. We recently had a very rare restart of the Linode - so that's why I needed to basically operate the machinery here. I'm back on the 1st now - so unfortunately after Tuesday next week. If you can investigate where else you might want to put it then when I come back we can do that together. Is that a possibility?

[chrismurrph]

Perhaps the essential first thing to do (whether or not it later goes onto another machine) is to have the server running standalone (so an init.d process in Linux) in its own JVM. The difference being that the server would start its own version of jetty rather than being a webapp housed in a web server container (again jetty) as it is now. That way it should at least be immune to GC slowdowns caused by other servers.

[angelsinistanbul]

please your object must be serialized..