ISG-ICS / cloudberry

Big Data Visualization
http://cloudberry.ics.uci.edu
91 stars 82 forks source link

Cloudberry: OOM issue #439

Closed waans11 closed 6 years ago

waans11 commented 6 years ago

After connecting to Twittermap demo and issuing a query, the frontend generated the following message: "WebSocket connection to ‘ws://cloudberry.ics.uci.edu/ws’ failed: Error during WebSocket handshake: Unexpected response code: 503"

The Cloudberry (neo module) log file says the following.

2017-11-27 17:26:37,706 [DEBUG] from application in ForkJoinPool-1-worker-8 - Query: select coll_count( (select value c from (select value t from twitter.ds_tweet_76af47488ac4ecce7c29005f15cf7d0e t) as c) ) as count; 2017-11-27 18:04:33,018 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16798 - Uncaught fatal error from thread [application-akka.actor.default-dispatcher-16788] shutting down ActorSystem [application] java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:114) at com.fasterxml.jackson.core.io.IOContext.allocConcatBuffer(IOContext.java:194) at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.(WriterBasedJsonGenerator.java:76) at com.fasterxml.jackson.core.JsonFactory._createGenerator(JsonFactory.java:1308) at com.fasterxml.jackson.core.JsonFactory.createGenerator(JsonFactory.java:1119) at play.api.libs.json.jackson.JacksonJson$.stringJsonGenerator(JacksonJson.scala:211) at play.api.libs.json.jackson.JacksonJson$.generateFromJsValue(JacksonJson.scala:233) at play.api.libs.json.Json$.stringify(Json.scala:61) at play.api.libs.json.JsValue$class.toString(JsValue.scala:16) at play.api.libs.json.JsObject.toString(JsValue.scala:76) at actor.RequestRouter$$anonfun$receive$1$$anonfun$applyOrElse$1.apply(RequestRouter.scala:26) at actor.RequestRouter$$anonfun$receive$1$$anonfun$applyOrElse$1.apply(RequestRouter.scala:26) at play.api.LoggerLike$class.info(Logger.scala:92) at play.api.Logger.info(Logger.scala:150) at actor.RequestRouter$$anonfun$receive$1.applyOrElse(RequestRouter.scala:26) at akka.actor.Actor$class.aroundReceive(Actor.scala:482) at actor.RequestRouter.aroundReceive(RequestRouter.scala:13) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2017-11-27 18:04:33,019 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16798 - Uncaught fatal error from thread [application-akka.actor.default-dispatcher-16812] shutting down ActorSystem [application] java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:114) at com.fasterxml.jackson.core.io.IOContext.allocConcatBuffer(IOContext.java:194) at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.(WriterBasedJsonGenerator.java:76) at com.fasterxml.jackson.core.JsonFactory._createGenerator(JsonFactory.java:1308) at com.fasterxml.jackson.core.JsonFactory.createGenerator(JsonFactory.java:1119) at play.api.libs.json.jackson.JacksonJson$.stringJsonGenerator(JacksonJson.scala:211) at play.api.libs.json.jackson.JacksonJson$.generateFromJsValue(JacksonJson.scala:233) at play.api.libs.json.Json$.stringify(Json.scala:61) at play.api.libs.json.JsValue$class.toString(JsValue.scala:16) at play.api.libs.json.JsObject.toString(JsValue.scala:76) at actor.RequestRouter$$anonfun$receive$1$$anonfun$applyOrElse$1.apply(RequestRouter.scala:26) at actor.RequestRouter$$anonfun$receive$1$$anonfun$applyOrElse$1.apply(RequestRouter.scala:26) at play.api.LoggerLike$class.info(Logger.scala:92) at play.api.Logger.info(Logger.scala:150) at actor.RequestRouter$$anonfun$receive$1.applyOrElse(RequestRouter.scala:26) at akka.actor.Actor$class.aroundReceive(Actor.scala:482) at actor.RequestRouter.aroundReceive(RequestRouter.scala:13) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2017-11-27 18:05:17,975 [DEBUG] from akka.stream.impl.StreamSupervisor in application-akka.actor.default-dispatcher-16798 - now supervising Actor[akka://application/user/StreamSupervisor-0/flow-2301-0-actorRefSource#-1131891654] 2017-11-27 18:05:17,975 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16798 - Uncaught fatal error from thread [application-akka.actor.default-dispatcher-16805] shutting down ActorSystem [application] java.lang.OutOfMemoryError: Java heap space 2017-11-27 18:05:17,975 [DEBUG] from akka.stream.impl.ActorRefSourceActor in application-akka.actor.default-dispatcher-16798 - started (akka.stream.impl.ActorRefSourceActor@18288263) 2017-11-27 18:05:17,975 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16798 - Uncaught fatal error from thread [application-akka.actor.default-dispatcher-16773] shutting down ActorSystem [application] java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:114) at com.fasterxml.jackson.core.io.IOContext.allocConcatBuffer(IOContext.java:194) at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.(WriterBasedJsonGenerator.java:76) at com.fasterxml.jackson.core.JsonFactory._createGenerator(JsonFactory.java:1308) at com.fasterxml.jackson.core.JsonFactory.createGenerator(JsonFactory.java:1119) at play.api.libs.json.jackson.JacksonJson$.stringJsonGenerator(JacksonJson.scala:211) at play.api.libs.json.jackson.JacksonJson$.generateFromJsValue(JacksonJson.scala:233) at play.api.libs.json.Json$.stringify(Json.scala:61) at play.api.libs.json.JsValue$class.toString(JsValue.scala:16) at play.api.libs.json.JsObject.toString(JsValue.scala:76) at actor.RequestRouter$$anonfun$receive$1$$anonfun$applyOrElse$1.apply(RequestRouter.scala:26) at actor.RequestRouter$$anonfun$receive$1$$anonfun$applyOrElse$1.apply(RequestRouter.scala:26) at play.api.LoggerLike$class.info(Logger.scala:92) at play.api.Logger.info(Logger.scala:150) at actor.RequestRouter$$anonfun$receive$1.applyOrElse(RequestRouter.scala:26) at akka.actor.Actor$class.aroundReceive(Actor.scala:482) at actor.RequestRouter.aroundReceive(RequestRouter.scala:13) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2017-11-27 18:05:17,975 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16798 - Uncaught fatal error from thread [application-akka.actor.default-dispatcher-16807] shutting down ActorSystem [application] java.lang.OutOfMemoryError: Java heap space at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:114) at com.fasterxml.jackson.core.io.IOContext.allocConcatBuffer(IOContext.java:194) at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.(WriterBasedJsonGenerator.java:76) at com.fasterxml.jackson.core.JsonFactory._createGenerator(JsonFactory.java:1308) at com.fasterxml.jackson.core.JsonFactory.createGenerator(JsonFactory.java:1119) at play.api.libs.json.jackson.JacksonJson$.stringJsonGenerator(JacksonJson.scala:211) at play.api.libs.json.jackson.JacksonJson$.generateFromJsValue(JacksonJson.scala:233) at play.api.libs.json.Json$.stringify(Json.scala:61) at play.api.libs.json.JsValue$class.toString(JsValue.scala:16) at play.api.libs.json.JsObject.toString(JsValue.scala:76) at actor.RequestRouter$$anonfun$receive$1$$anonfun$applyOrElse$1.apply(RequestRouter.scala:26) at actor.RequestRouter$$anonfun$receive$1$$anonfun$applyOrElse$1.apply(RequestRouter.scala:26) at play.api.LoggerLike$class.info(Logger.scala:92) at play.api.Logger.info(Logger.scala:150) at actor.RequestRouter$$anonfun$receive$1.applyOrElse(RequestRouter.scala:26) at akka.actor.Actor$class.aroundReceive(Actor.scala:482) at actor.RequestRouter.aroundReceive(RequestRouter.scala:13) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) at akka.actor.ActorCell.invoke(ActorCell.scala:495) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) at akka.dispatch.Mailbox.run(Mailbox.scala:224) at akka.dispatch.Mailbox.exec(Mailbox.scala:234) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2017-11-27 18:05:17,976 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16798 - Uncaught fatal error from thread [application-akka.actor.default-dispatcher-16809] shutting down ActorSystem [application] java.lang.OutOfMemoryError: Java heap space 2017-11-27 18:06:07,470 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16773 - Uncaught fatal error from thread [application-akka.actor.default-dispatcher-16800] shutting down ActorSystem [application] java.lang.OutOfMemoryError: Java heap space 2017-11-27 18:06:46,825 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16798 - Uncaught fatal error from thread [application-akka.actor.default-dispatcher-16804] shutting down ActorSystem [application] java.lang.OutOfMemoryError: Java heap space 2017-11-27 18:09:43,128 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16812 - no longer watched by Actor[akka://application/user/$1#256981433] 2017-11-27 18:10:22,516 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16812 - stopping 2017-11-27 18:10:22,517 [DEBUG] from edu.uci.ics.cloudberry.zion.actor.BerryClient in application-akka.actor.default-dispatcher-16807 - stopped 2017-11-27 18:10:22,517 [DEBUG] from edu.uci.ics.cloudberry.zion.actor.BerryClient in application-akka.actor.default-dispatcher-16807 - stopped 2017-11-27 18:10:22,517 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16807 - no longer watched by Actor[akka://application/user/$Fg#1125731500] 2017-11-27 18:12:26,651 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16807 - stopping 2017-11-27 18:12:26,651 [DEBUG] from edu.uci.ics.cloudberry.zion.actor.BerryClient in application-akka.actor.default-dispatcher-16807 - stopped 2017-11-27 18:12:26,651 [DEBUG] from edu.uci.ics.cloudberry.zion.actor.BerryClient in application-akka.actor.default-dispatcher-16807 - stopped 2017-11-27 18:12:26,653 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16780 - stopped 2017-11-27 18:12:26,653 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16780 - stopped 2017-11-27 18:13:58,721 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16788 - no longer watched by Actor[akka://application/user/$X#-2001594988] 2017-11-27 18:13:58,721 [DEBUG] from akka.stream.impl.ActorRefSinkActor in application-akka.actor.default-dispatcher-16788 - received AutoReceiveMessage Envelope(Terminated(Actor[akka://application/user/$Fg#1125731500]),Actor[akka://application/user/$Fg#1125731500]) 2017-11-27 18:13:58,721 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16788 - stopping 2017-11-27 18:14:40,180 [DEBUG] from edu.uci.ics.cloudberry.zion.actor.BerryClient in application-akka.actor.default-dispatcher-16788 - stopped 2017-11-27 18:14:40,180 [DEBUG] from akka.actor.LocalActorRefProvider$Guardian in application-akka.actor.default-dispatcher-16788 - stopping 2017-11-27 18:14:40,180 [DEBUG] from controllers.Cloudberry$Listener in application-akka.actor.default-dispatcher-16788 - stopped 2017-11-27 18:14:40,180 [DEBUG] from akka.stream.impl.ActorRefSinkActor in application-akka.actor.default-dispatcher-16788 - received AutoReceiveMessage Envelope(Terminated(Actor[akka://application/user/$1#256981433]),Actor[akka://application/user/$1#256981433]) 2017-11-27 18:14:40,180 [DEBUG] from akka.event.EventStreamUnsubscriber in application-akka.actor.default-dispatcher-16788 - received AutoReceiveMessage Envelope(Terminated(Actor[akka://application/user/$b#-542715220]),Actor[akka://application/user/$b#-542715220]) 2017-11-27 18:23:39,738 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16799 - no longer watched by Actor[akka://application/user/$yc#-1771662942] 2017-11-27 18:23:45,661 [DEBUG] from akka.stream.impl.ActorRefSourceActor in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:23:45,662 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16812 - stopping 2017-11-27 18:24:28,408 [DEBUG] from akka.stream.impl.ActorRefSourceActor in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:24:34,318 [DEBUG] from edu.uci.ics.cloudberry.zion.actor.BerryClient in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:24:45,829 [DEBUG] from edu.uci.ics.cloudberry.zion.actor.BerryClient in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:24:57,718 [DEBUG] from akka.stream.impl.ActorRefSourceActor in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:37:26,429 [DEBUG] from edu.uci.ics.cloudberry.zion.actor.BerryClient in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:37:44,107 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:45:47,794 [DEBUG] from akka.stream.impl.ActorRefSinkActor in application-akka.actor.default-dispatcher-16812 - received AutoReceiveMessage Envelope(Terminated(Actor[akka://application/user/$yc#-1771662942]),Actor[akka://application/user/$yc#-1771662942]) 2017-11-27 18:45:53,730 [DEBUG] from akka.stream.impl.ActorRefSourceActor in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:46:43,328 [DEBUG] from akka.stream.impl.ActorRefSourceActor in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:53:17,473 [DEBUG] from actor.RequestRouter in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:53:17,473 [DEBUG] from akka.stream.impl.ActorRefSourceActor in application-akka.actor.default-dispatcher-16812 - stopped 2017-11-27 18:53:17,474 [DEBUG] from akka.stream.impl.StreamSupervisor in application-akka.actor.default-dispatcher-16812 - stopping 2017-11-27 19:26:33,553 [ERROR] from play.core.server.netty.PlayRequestHandler in netty-event-loop-20 - Exception caught in Netty java.lang.OutOfMemoryError: Java heap space

JavierJia commented 6 years ago

This is caused by the logger.