search

VirtualFlyBrain / VFB2

Virtual Fly Brain Documentation Site
https://virtualflybrain.org
MIT License
2 stars 6 forks source link

Virgo/Geppetto Java Memory Issue #290

Open Robbie1977 opened 3 years ago

Robbie1977 commented 3 years ago

Virgo is after approx 24 hrs is filling up all available heap space and then crashing with this error:

[2020-11-25 09:26:18.358] ERROR http-nio-8080-exec-7         System.err                                                        Exception in thread "http-nio-8080-exec-7" java.lang.OutOfMemoryError: Java heap space
[2020-11-25 09:26:18.360] ERROR http-nio-8080-exec-7         System.err                                                         at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
[2020-11-25 09:26:18.360] ERROR http-nio-8080-exec-7         System.err                                                         at java.nio.CharBuffer.allocate(CharBuffer.java:335)
[2020-11-25 09:26:18.360] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.websocket.WsFrameBase.processInitialHeader(WsFrameBase.java:196)
[2020-11-25 09:26:18.360] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.websocket.WsFrameBase.processInputBuffer(WsFrameBase.java:118)
[2020-11-25 09:26:18.361] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:82)
[2020-11-25 09:26:18.361] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.websocket.server.WsFrameServer.doOnDataAvailable(WsFrameServer.java:171)
[2020-11-25 09:26:18.361] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.websocket.server.WsFrameServer.notifyDataAvailable(WsFrameServer.java:151)
[2020-11-25 09:26:20.362] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:148)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
[2020-11-25 09:26:20.363] ERROR http-nio-8080-exec-7         System.err                                                         at java.lang.Thread.run(Thread.java:745)

@ddelpiano How often are we clearing out the user sessions?

Support info: Wed Nov 25 2020 16:20:23 GMT+0000 (Greenwich Mean Time) Chrome 86.0.4240.198 [1920,1000] https://v2.virtualflybrain.org/org.geppetto.frontend/geppetto?id=Lee2020&i=VFB_00101567

Robbie1977 commented 3 years ago

FYI The last WebSocket mention before the crash is Websocket connection removed Connection509

I'm catering the whole log for any crash so we can see if this is consistent on the next occurrence.

Robbie1977 commented 3 years ago 

[2020-11-25 20:54:40.535] INFO  http-nio-8080-exec-7         org.geppetto.frontend.controllers.ConnectionsManager              Websocket connection removed Connection565
[2020-11-25 20:54:40.536] INFO  http-nio-8080-exec-7         org.geppetto.frontend.controllers.WebsocketConnection             Closed Connection ...234
[2020-11-25 20:54:45.771] INFO  http-nio-8080-exec-4         org.geppetto.frontend.controllers.WebsocketConnection             Session Binary size >> 9999999
[2020-11-25 20:54:45.772] INFO  http-nio-8080-exec-4         org.geppetto.frontend.controllers.WebsocketConnection             Session Text size >> 9999999
[2020-11-25 20:54:45.772] INFO  http-nio-8080-exec-4         org.geppetto.frontend.messaging.DefaultMessageSender              Initializing message sender - queuing: true, compression: true, discard messages if queues full: false
[2020-11-25 20:54:45.774] INFO  http-nio-8080-exec-4         org.geppetto.frontend.messaging.DefaultMessageSender              Sending message took : 1 ms
[2020-11-25 20:54:45.774] INFO  http-nio-8080-exec-4         org.geppetto.frontend.controllers.ConnectionsManager              New websocket connection null
[2020-11-25 20:54:45.775] INFO  http-nio-8080-exec-4         org.geppetto.frontend.messaging.DefaultMessageSender              Sending message took : 1 ms
[2020-11-25 20:54:45.775] INFO  http-nio-8080-exec-4         org.geppetto.frontend.controllers.WebsocketConnection             Open Connection ...235
[2020-11-25 20:54:49.472] ERROR http-nio-8080-exec-5         System.err                                                        Exception in thread "http-nio-8080-exec-5" java.lang.OutOfMemoryError: Java heap space
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at java.nio.CharBuffer.allocate(CharBuffer.java:335)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.websocket.WsFrameBase.processInitialHeader(WsFrameBase.java:196)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.websocket.WsFrameBase.processInputBuffer(WsFrameBase.java:118)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:82)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.websocket.server.WsFrameServer.doOnDataAvailable(WsFrameServer.java:171)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.websocket.server.WsFrameServer.notifyDataAvailable(WsFrameServer.java:151)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:148)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:53)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
[2020-11-25 20:54:49.474] ERROR http-nio-8080-exec-5         System.err                                                         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[2020-11-25 20:54:49.475] ERROR http-nio-8080-exec-5         System.err                                                         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[2020-11-25 20:54:49.475] ERROR http-nio-8080-exec-5         System.err                                                         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
[2020-11-25 20:54:49.475] ERROR http-nio-8080-exec-5         System.err                                                         at java.lang.Thread.run(Thread.java:745)```