search

TheHive-Project / TheHive

TheHive: a Scalable, Open Source and Free Security Incident Response Platform
https://thehive-project.org
GNU Affero General Public License v3.0
3.43k stars 622 forks source link

the hive crashes due to Java heap space #552

Open garanews opened 6 years ago

garanews commented 6 years ago

Request Type

Bug

Work Environment

Question Answer
OS version (server) Ubuntu 16.04.3 LTS
TheHive version / git hash 3.0.5
Package Type DEB

Problem Description

The hive service crashes regularly due to Java heap space, don't know if related to MISP sync. Current RAM settings: ES 12GB TheHive 8GB No webhooks configured.

tail -f thehive_err.log

       at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:653)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
        at akka.actor.ActorCell.invoke(ActorCell.scala:496)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
Uncaught error from thread [application-akka.actor.default-dispatcher-2]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for for ActorSystem[application]
java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding.decode(StringCoding.java:215)
        at java.lang.String.<init>(String.java:463)
        at java.lang.String.<init>(String.java:515)
        at play.shaded.ahc.org.asynchttpclient.netty.NettyResponse.getResponseBody(NettyResponse.java:204)
        at play.api.libs.ws.ahc.AhcWSUtils$.getResponseBody(AhcWSUtils.scala:13)
        at play.api.libs.ws.ahc.StandaloneAhcWSResponse.body$lzycompute(StandaloneAhcWSResponse.scala:36)
        at play.api.libs.ws.ahc.StandaloneAhcWSResponse.body(StandaloneAhcWSResponse.scala:35)
        at play.api.libs.ws.ahc.AhcWSResponse.body(AhcWSResponse.scala:64)
        at connectors.misp.MispSrv.$anonfun$getAttributesFromMisp$3(MispSrv.scala:115)
        at connectors.misp.MispSrv$$Lambda$1894/1166238033.apply(Unknown Source)
        at scala.util.Success.$anonfun$map$1(Try.scala:251)
        at scala.util.Success.map(Try.scala:209)
        at scala.concurrent.Future.$anonfun$map$1(Future.scala:289)
        at scala.concurrent.Future$$Lambda$1255/358362062.apply(Unknown Source)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
        at scala.concurrent.impl.Promise$$Lambda$235/1998397435.apply(Unknown Source)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$1284/1062632924.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

tail -f thehive.log

   excluded tags:  Type:OSINTtlp:white
[info] i.n.u.i.PlatformDependent - Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
[info] c.m.MispSynchro - Update of MISP events is starting ...
[info] c.c.s.CortexClient - new Cortex(CORTEX-SERVER-ID, http://127.0.0.1:9001) authentication: no
[info] c.c.s.CortexSrv - Search for unfinished job ...
[info] c.c.s.CortexSrv - 0 jobs found
[info] c.m.MispSynchro - Synchronize MISP MISP-CYBER from Some(Thu Apr 19 18:32:20 UTC 2018)
[info] play.api.Play - Application started (Prod)
[info] p.c.s.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
[info] c.m.MispSynchro - Update of MISP events is starting ...
[info] c.m.MispSynchro - Synchronize MISP MISP-CYBER from Some(Thu Apr 19 18:32:20 UTC 2018)
[info] o.e.ErrorHandler - POST /api/alert returned 400
org.elastic4play.ConflictError: [alert][4eda23fef341cd3b597531e4e2aa043a]: version conflict, document already exists (current version [1])
        at org.elastic4play.database.DBCreate.convertError(DBCreate.scala:81)
        at org.elastic4play.database.DBCreate.convertError(DBCreate.scala:80)
        at org.elastic4play.database.DBCreate.$anonfun$apply$10(DBCreate.scala:76)
        at scala.concurrent.Future.$anonfun$transform$3(Future.scala:242)
        at scala.concurrent.Future.$anonfun$transform$1(Future.scala:242)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
[error] s.DeadLetterMonitoringActor - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-UKljao9dRW#120905685],Actor[akka://application/deadLetters])
[info] o.e.ErrorHandler - GET /api/stream/UKljao9dRW returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-UKljao9dRW)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
[error] s.DeadLetterMonitoringActor - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-hD7jH9KkXf#-307781972],Actor[akka://application/deadLetters])
[info] o.e.ErrorHandler - GET /api/stream/hD7jH9KkXf returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-hD7jH9KkXf)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
[error] s.StreamActor - No request to submit !
[error] s.DeadLetterMonitoringActor - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-w7ZNlgOwfo#1804790092],Actor[akka://application/deadLetters])
[info] o.e.ErrorHandler - GET /api/stream/w7ZNlgOwfo returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-w7ZNlgOwfo)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
[error] s.StreamActor - No request to submit !
[error] s.DeadLetterMonitoringActor - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-TKUF6TsnvY#-1770668607],Actor[akka://application/deadLetters])
[info] o.e.ErrorHandler - GET /api/stream/TKUF6TsnvY returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-TKUF6TsnvY)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
[warn] a.a.ActorSystemImpl - Explicitly set HTTP header 'Content-Type: application/zip' is ignored, explicit `Content-Type` header is not allowed. Set `HttpResponse.entity.contentType` instead.
[warn] a.a.ActorSystemImpl - Explicitly set HTTP header 'Content-Length: 2319' is ignored, explicit `Content-Length` header is not allowed. Use the appropriate HttpEntity subtype.
[error] s.DeadLetterMonitoringActor - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-531ZmadfEE#492633680],Actor[akka://application/deadLetters])
[info] o.e.ErrorHandler - GET /api/stream/531ZmadfEE returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-531ZmadfEE)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
[error] s.StreamActor - No request to submit !
[info] c.m.MispSynchro - Update of MISP events is starting ...
[info] c.m.MispSynchro - Synchronize MISP MISP-CYBERRESPONSE from Some(Thu Apr 19 18:32:20 UTC 2018)
[info] o.e.ErrorHandler - POST /api/connector/misp/export/AWK4r-7RXPcIR2mAht86/MISP-CYBER returned 400
org.elastic4play.AttributeCheckingError: [Attribute data is read-only][Attribute dataType is read-only][Attribute message is read-only][Attribute startDate is read-only][Attribute attachment is read-only][Attribute tlp is read-only][Attribute tags is read-only][Attribute ioc is read-only][Attribute data is read-only][Attribute dataType is read-only][Attribute message is read-only][Attribute startDate is read-only][Attribute attachment is read-only][Attribute tlp is read-only][Attribute tags is read-only][Attribute ioc is read-only][Attribute data is read-only][Attribute dataType is read-only][Attribute message is read-only][Attribute startDate is read-only][Attribute attachment is read-only][Attribute tlp is read-only][Attribute tags is read-only][Attribute ioc is read-only][Attribute data is read-only][Attribute dataType is read-only][Attribute message is read-only][Attribute startDate is read-only][Attribute attachment is read-only][Attribute tlp is read-only][Attribute tags is read-only][Attribute ioc is read-only][Attribute data is read-only][Attribute dataType is read-only][Attribute message is read-only][Attribute startDate is read-only][Attribute attachment is read-only][Attribute tlp is read-only][Attribute tags is read-only][Attribute ioc is read-only][Attribute data is read-only][Attribute dataType is read-only][Attribute message is read-only][Attribute startDate is read-only][Attribute attachment is read-only][Attribute tlp is read-only][Attribute tags is read-only][Attribute ioc is read-only][Attribute data is read-only][Attribute dataType is read-only][Attribute message is read-only][Attribute startDate is read-only][Attribute attachment is read-only][Attribute tlp is read-only][Attribute tags is read-only][Attribute ioc is read-only]
        at org.elastic4play.services.UpdateSrv.$anonfun$checkAttributes$5(UpdateSrv.scala:46)
        at org.scalactic.Bad.fold(Or.scala:1387)
        at org.elastic4play.services.UpdateSrv.checkAttributes(UpdateSrv.scala:46)
        at org.elastic4play.services.UpdateSrv.$anonfun$doUpdate$1(UpdateSrv.scala:52)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:304)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[info] p.c.s.AkkaHttpServer - Stopping server...
[error] s.DeadLetterMonitoringActor - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-MHxe6mdXho#-1715506365],Actor[akka://application/deadLetters])
[info] o.e.ErrorHandler - GET /api/stream/MHxe6mdXho returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-MHxe6mdXho)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
[error] a.a.ActorSystemImpl - Uncaught error from thread [application-akka.actor.default-dispatcher-2]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]
java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding.decode(StringCoding.java:215)
        at java.lang.String.<init>(String.java:463)
        at java.lang.String.<init>(String.java:515)
        at play.shaded.ahc.org.asynchttpclient.netty.NettyResponse.getResponseBody(NettyResponse.java:204)
        at play.api.libs.ws.ahc.AhcWSUtils$.getResponseBody(AhcWSUtils.scala:13)
        at play.api.libs.ws.ahc.StandaloneAhcWSResponse.body$lzycompute(StandaloneAhcWSResponse.scala:36)
        at play.api.libs.ws.ahc.StandaloneAhcWSResponse.body(StandaloneAhcWSResponse.scala:35)
        at play.api.libs.ws.ahc.AhcWSResponse.body(AhcWSResponse.scala:64)
        at connectors.misp.MispSrv.$anonfun$getAttributesFromMisp$3(MispSrv.scala:115)
        at connectors.misp.MispSrv$$Lambda$1894/1166238033.apply(Unknown Source)
[info] a.a.CoordinatedShutdown - Starting coordinated shutdown from JVM shutdown hook
[warn] a.a.CoordinatedShutdown - CoordinatedShutdown from JVM shutdown failed: Futures timed out after [10000 milliseconds]
[info] c.m.MispSynchro - Misp synchronization failed
akka.stream.AbruptStageTerminationException: GraphStage [akka.stream.impl.SeqStage$$anon$4@6b5e06c7] terminated abruptly, caused by for example materializer or actor system termination.
[info] c.m.MispSynchro - Misp synchronization failed
akka.stream.AbruptStageTerminationException: GraphStage [akka.stream.impl.SeqStage$$anon$4@314e5870] terminated abruptly, caused by for example materializer or actor system termination.
[info] c.m.MispSynchro - Misp synchronization failed
akka.stream.AbruptStageTerminationException: GraphStage [akka.stream.impl.SeqStage$$anon$4@efb9606] terminated abruptly, caused by for example materializer or actor system termination.
[warn] a.a.CoordinatedShutdown - Coordinated shutdown phase [actor-system-terminate] timed out after 10000 milliseconds
[info] c.m.MispSynchro - Stopping MISP fetching ...
[info] o.r.Reflections - Reflections took 429 ms to scan 5 urls, producing 115 keys and 1153 values

tail -f application.log

        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)
2018-04-20 09:53:05,421 [ERROR] from services.DeadLetterMonitoringActor in application-akka.actor.default-dispatcher-6 - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-w8chAvicUo#1398614848],Actor[akka://application/deadLetters])
2018-04-20 09:53:05,423 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-27 - GET /api/stream/w8chAvicUo returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-w8chAvicUo)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)
2018-04-20 09:54:02,345 [ERROR] from services.DeadLetterMonitoringActor in application-akka.actor.default-dispatcher-26 - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-bUieVs9ZbL#1735276249],Actor[akka://application/deadLetters])
2018-04-20 09:54:02,346 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-8 - GET /api/stream/bUieVs9ZbL returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-bUieVs9ZbL)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)
2018-04-20 09:54:22,837 [ERROR] from services.DeadLetterMonitoringActor in application-akka.actor.default-dispatcher-26 - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-261MWB2pZH#1954417811],Actor[akka://application/deadLetters])
2018-04-20 09:54:30,610 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-50 - GET /api/stream/261MWB2pZH returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-261MWB2pZH)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)
2018-04-20 09:55:34,193 [ERROR] from services.DeadLetterMonitoringActor in application-akka.actor.default-dispatcher-50 - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-DpjBrEOwMy#141893177],Actor[akka://application/deadLetters])
2018-04-20 09:55:34,196 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-50 - GET /api/stream/DpjBrEOwMy returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-DpjBrEOwMy)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)
2018-04-20 09:55:34,199 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-50 - GET /api/stream/sxQEkM5yvN returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-sxQEkM5yvN)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)
2018-04-20 09:55:34,199 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-50 - GET /api/stream/qcMu0BYE59 returned 500
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://application/), Path(/user/stream-qcMu0BYE59)]] after [62000 ms]. Sender[null] sent message of type "services.StreamActor$GetOperations$".
        at akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:601)
        at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:140)
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:866)
        at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109)
        at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:864)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)
2018-04-20 09:55:34,204 [ERROR] from services.DeadLetterMonitoringActor in application-akka.actor.default-dispatcher-50 - receive dead message : DeadLetter(Commit(0096557015e22d52:-1ab321:162e23ec6ce:-8000:520),Actor[akka://application/deadLetters],Actor[akka://application/user/stream-MV28f7RyM7#1306075654])
2018-04-20 09:55:34,204 [ERROR] from services.DeadLetterMonitoringActor in application-akka.actor.default-dispatcher-50 - receive dead message : DeadLetter(Commit(0096557015e22d52:-1ab321:162e23ec6ce:-8000:521),Actor[akka://application/deadLetters],Actor[akka://application/user/stream-MV28f7RyM7#1306075654])
2018-04-20 09:55:34,204 [ERROR] from services.DeadLetterMonitoringActor in application-akka.actor.default-dispatcher-50 - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-qcMu0BYE59#356533275],Actor[akka://application/deadLetters])
2018-04-20 09:55:34,204 [ERROR] from services.DeadLetterMonitoringActor in application-akka.actor.default-dispatcher-50 - receive dead message : DeadLetter(StreamMessages(List()),Actor[akka://application/user/stream-sxQEkM5yvN#1937974704],Actor[akka://application/deadLetters])
2018-04-20 09:55:34,286 [INFO] from play.core.server.AkkaHttpServer in Thread-7 - Stopping server...
2018-04-20 09:55:34,287 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-2 - Uncaught error from thread [application-akka.actor.default-dispatcher-4]: GC overhead limit exceeded, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at scala.collection.immutable.List.$colon$colon(List.scala:112)
        at scala.collection.immutable.List.$plus$colon(List.scala:210)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:164)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:128)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:123)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3786)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2115)
        at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:235)
        at play.api.libs.json.StaticBinding$.parseJsValue(StaticBinding.scala:16)
        at play.api.libs.json.Json$.parse(Json.scala:171)
        at org.elastic4play.database.DBUtils$.hit2json(DBUtils.scala:35)
        at org.elastic4play.database.DBFind.$anonfun$apply$2(DBFind.scala:114)
        at org.elastic4play.database.DBFind$$Lambda$1446/1348426602.apply(Unknown Source)
        at akka.stream.impl.fusing.Map$$anon$7.onPush(Ops.scala:47)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:499)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:401)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:571)
        at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:457)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:546)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:728)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:743)
        at akka.actor.Actor.aroundReceive(Actor.scala:517)
        at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:653)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
        at akka.actor.ActorCell.invoke(ActorCell.scala:496)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
2018-04-20 09:55:34,289 [INFO] from akka.actor.CoordinatedShutdown in application-akka.actor.default-dispatcher-2 - Starting coordinated shutdown from JVM shutdown hook
2018-04-20 09:55:34,324 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-2 - Misp synchronization failed
akka.stream.AbruptStageTerminationException: GraphStage [akka.stream.impl.SeqStage$$anon$4@770e542a] terminated abruptly, caused by for example materializer or actor system termination.
2018-04-20 09:55:34,325 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-8 - Misp synchronization failed
akka.stream.AbruptStageTerminationException: GraphStage [akka.stream.impl.SeqStage$$anon$4@623f3994] terminated abruptly, caused by for example materializer or actor system termination.
2018-04-20 09:55:34,331 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-47 - Misp synchronization failed
akka.stream.AbruptStageTerminationException: GraphStage [akka.stream.impl.SeqStage$$anon$4@5e66b0a6] terminated abruptly, caused by for example materializer or actor system termination.
2018-04-20 09:55:34,397 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-8 - Update of MISP events is starting ...
2018-04-20 09:55:34,397 [INFO] from connectors.misp.MispSynchro in Thread-7 - Stopping MISP fetching ...
2018-04-20 09:55:45,680 [INFO] from org.reflections.Reflections in main - Reflections took 376 ms to scan 5 urls, producing 115 keys and 1153 values
2018-04-20 09:55:45,757 [INFO] from module in main - Loading model class models.DashboardModel
2018-04-20 09:55:45,759 [INFO] from module in main - Loading model class org.elastic4play.services.DBListModel
2018-04-20 09:55:45,760 [INFO] from module in main - Loading model class models.TaskModel
2018-04-20 09:55:45,760 [INFO] from module in main - Loading model class models.AlertModel
2018-04-20 09:55:45,761 [INFO] from module in main - Loading model class org.elastic4play.services.AttachmentModel
2018-04-20 09:55:45,762 [INFO] from module in main - Loading model class models.LogModel
2018-04-20 09:55:45,762 [INFO] from module in main - Loading model class models.ArtifactModel
2018-04-20 09:55:45,763 [INFO] from module in main - Loading model class connectors.cortex.models.ReportTemplateModel
2018-04-20 09:55:45,763 [INFO] from module in main - Loading model class models.UserModel
2018-04-20 09:55:45,764 [INFO] from module in main - Loading model class connectors.cortex.models.JobModel
jeffrey-e commented 6 years ago

Hi guys, seeing more or less the same after upgrading to 3.1.2-1. It seems it slowly runs OOM within a day. I thought it might be due to Cortex now also querying Elasticsearch, but during the moment of crash Cortex shouldn't have anything to do. I hope you guys can help us, because the environment breaks after less than a day after being started. See stack traces below...

2018-10-19 23:28:17,918 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-17 - Update of MISP events is starting ...
2018-10-19 23:28:18,383 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-16 - Synchronize MISP MISP-extern from Some(Wed Oct 17 15:52:08 CEST 2018)
2018-10-19 23:29:02,955 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-15 - Misp synchronization failed
java.net.ConnectException: handshake timed out
        at play.shaded.ahc.org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:168)
        at play.shaded.ahc.org.asynchttpclient.netty.channel.NettyConnectListener$1.onFailure(NettyConnectListener.java:139)
        at play.shaded.ahc.org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:26)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
        at play.shaded.ahc.io.netty.handler.ssl.SslHandler.notifyHandshakeFailure(SslHandler.java:1443)
        at play.shaded.ahc.io.netty.handler.ssl.SslHandler.access$1100(SslHandler.java:161)
        at play.shaded.ahc.io.netty.handler.ssl.SslHandler$5.run(SslHandler.java:1613)
        at play.shaded.ahc.io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at play.shaded.ahc.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
        at play.shaded.ahc.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
        at play.shaded.ahc.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLException: handshake timed out
        at play.shaded.ahc.io.netty.handler.ssl.SslHandler.handshake(...)(Unknown Source)
2018-10-19 23:29:15,917 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-18 - Update of MISP events is starting ...
2018-10-19 23:29:15,918 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-14 - Synchronize MISP MISP-extern from Some(Wed Oct 17 15:52:08 CEST 2018)
2018-10-19 23:29:16,174 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-19 - Misp synchronization completed
2018-10-19 23:30:17,922 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-2 - Update of MISP events is starting ...
2018-10-19 23:30:18,361 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-14 - Synchronize MISP MISP-extern from Some(Wed Oct 17 15:52:08 CEST 2018)
2018-10-19 23:31:17,694 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-18 - Misp synchronization completed
2018-10-19 23:32:12,043 [ERROR] from akka.dispatch.TaskInvocation in application-akka.actor.default-dispatcher-17 - Futures timed out after [10 seconds]
java.util.concurrent.TimeoutException: Futures timed out after [10 seconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:255)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
        at scala.concurrent.Await$.$anonfun$result$1(package.scala:215)
        at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:167)
        at akka.dispatch.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3641)
        at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:165)
        at scala.concurrent.Await$.result(package.scala:142)
        at com.sksamuel.elastic4s.ElasticApi$RichFuture.await(ElasticApi.scala:79)
        at org.elastic4play.database.DBIndex.$anonfun$indexStatus$1(DBIndex.scala:100)
        at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:12)
        at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:167)
        at akka.dispatch.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3641)
        at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:165)
        at scala.concurrent.package$.blocking(package.scala:142)
        at org.elastic4play.database.DBIndex.indexStatus(DBIndex.scala:100)
        at org.elastic4play.services.MigrationSrv.isReady(MigrationSrv.scala:177)
        at connectors.misp.MispSynchro.$anonfun$initScheduler$1(MispSynchro.scala:48)
        at akka.actor.Scheduler$$anon$2.run(Scheduler.scala:83)
        at akka.actor.LightArrayRevolverScheduler$$anon$2$$anon$1.run(LightArrayRevolverScheduler.scala:101)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-10-19 23:40:03,858 [INFO] from org.elastic4play.ErrorHandler in application-akka.actor.default-dispatcher-14 - POST /api/alert returned 400
org.elastic4play.ConflictError: [alert][ac4497dea354bafef44364d20b754f2c]: version conflict, document already exists (current version [2])
        at org.elastic4play.database.DBCreate.convertError(DBCreate.scala:81)
        at org.elastic4play.database.DBCreate.convertError(DBCreate.scala:80)
        at org.elastic4play.database.DBCreate.$anonfun$apply$10(DBCreate.scala:76)
        at scala.concurrent.Future.$anonfun$transform$3(Future.scala:241)
        at scala.concurrent.Future.$anonfun$transform$1(Future.scala:241)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-10-19 23:57:38,642 [INFO] from org.elasticsearch.client.transport.TransportClientNodesService in elasticsearch[_client_][generic][T#1] - failed to get node info for {#transport#-1}{mTnQJOu6TSuJ8Oryp4q7jQ}{127.0.0.1}{127.0.0.1:9300}, disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][127.0.0.1:9300][cluster:monitor/nodes/liveness] request_id [257447] timed out after [14012ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-10-19 23:57:39,540 [WARN] from org.elasticsearch.transport.TransportService in elasticsearch[_client_][transport_client_boss][T#4] - Received response for a request that has timed out, sent [45242ms] ago, timed out [31230ms] ago, action [cluster:monitor/nodes/liveness], node [{#transport#-1}{mTnQJOu6TSuJ8Oryp4q7jQ}{127.0.0.1}{127.0.0.1:9300}], id [257447]
2018-10-19 23:58:16,536 [WARN] from play.shaded.ahc.io.netty.util.concurrent.DefaultPromise in AsyncHttpClient-8-11 - An exception was thrown by play.shaded.ahc.org.asynchttpclient.netty.request.NettyChannelConnector$1.operationComplete()
java.lang.OutOfMemoryError: Java heap space
        at sun.security.ssl.InputRecord.<init>(InputRecord.java:93)
        at sun.security.ssl.EngineInputRecord.<init>(EngineInputRecord.java:63)
        at sun.security.ssl.SSLEngineImpl.init(SSLEngineImpl.java:426)
        at sun.security.ssl.SSLEngineImpl.<init>(SSLEngineImpl.java:357)
        at sun.security.ssl.SSLContextImpl$AbstractTLSContext.createSSLEngineImpl(SSLContextImpl.java:465)
        at sun.security.ssl.SSLContextImpl.engineCreateSSLEngine(SSLContextImpl.java:203)
        at javax.net.ssl.SSLContext.createSSLEngine(SSLContext.java:361)
        at play.shaded.ahc.org.asynchttpclient.netty.ssl.JsseSslEngineFactory.newSslEngine(JsseSslEngineFactory.java:31)
        at play.shaded.ahc.org.asynchttpclient.netty.channel.ChannelManager.createSslHandler(ChannelManager.java:398)
        at play.shaded.ahc.org.asynchttpclient.netty.channel.ChannelManager.addSslHandler(ChannelManager.java:449)
        at play.shaded.ahc.org.asynchttpclient.netty.channel.NettyConnectListener.onSuccess(NettyConnectListener.java:115)
        at play.shaded.ahc.org.asynchttpclient.netty.request.NettyChannelConnector$1.onSuccess(NettyChannelConnector.java:92)
        at play.shaded.ahc.org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:26)
        at play.shaded.ahc.org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
        at play.shaded.ahc.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
        at play.shaded.ahc.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:257)
        at play.shaded.ahc.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:292)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
        at play.shaded.ahc.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
2018-10-19 23:58:36,658 [WARN] from play.shaded.ahc.io.netty.util.HashedWheelTimer in pool-3-thread-1 - An exception was thrown by TimerTask.
java.lang.OutOfMemoryError: Java heap space
2018-10-19 23:59:24,362 [WARN] from org.jboss.netty.channel.socket.nio.AbstractNioSelector in New I/O boss #3 - Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap$KeySet.iterator(HashMap.java:917)
        at java.util.HashSet.iterator(HashSet.java:173)
        at java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1039)
        at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1038)
        at org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:118)
        at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-10-20 00:00:04,450 [WARN] from play.shaded.ahc.io.netty.channel.DefaultChannelPipeline in AsyncHttpClient-6-14 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
play.shaded.ahc.io.netty.channel.ChannelPipelineException: play.shaded.ahc.org.asynchttpclient.netty.channel.ChannelManager$2.handlerAdded() has thrown an exception; removed.
        at play.shaded.ahc.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:610)
        at play.shaded.ahc.io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:44)
        at play.shaded.ahc.io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1355)
        at play.shaded.ahc.io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1090)
        at play.shaded.ahc.io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:640)
        at play.shaded.ahc.io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:456)
        at play.shaded.ahc.io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:378)
        at play.shaded.ahc.io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:428)
        at play.shaded.ahc.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
        at play.shaded.ahc.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
        at play.shaded.ahc.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at play.shaded.ahc.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
        at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
        at play.shaded.ahc.io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:111)
        at play.shaded.ahc.io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:105)
        at play.shaded.ahc.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:590)
        ... 12 common frames omitted
2018-10-20 00:02:34,781 [INFO] from org.elasticsearch.client.transport.TransportClientNodesService in elasticsearch[_client_][generic][T#4] - failed to get node info for {#transport#-1}{mTnQJOu6TSuJ8Oryp4q7jQ}{127.0.0.1}{127.0.0.1:9300}, disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][127.0.0.1:9300][cluster:monitor/nodes/liveness] request_id [257449] timed out after [27350ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-10-20 00:03:16,379 [WARN] from org.elasticsearch.transport.TransportService in elasticsearch[_client_][transport_client_boss][T#5] - Received response for a request that has timed out, sent [88255ms] ago, timed out [60905ms] ago, action [cluster:monitor/nodes/liveness], node [{#transport#-1}{mTnQJOu6TSuJ8Oryp4q7jQ}{127.0.0.1}{127.0.0.1:9300}], id [257449]
2018-10-20 00:06:16,865 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-16 - Uncaught error from thread [application-akka.actor.default-dispatcher-14]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]
java.lang.OutOfMemoryError: Java heap space
        at akka.dispatch.Envelope$.apply(AbstractDispatcher.scala:28)
        at akka.actor.Cell.sendMessage(ActorCell.scala:354)
        at akka.actor.Cell.sendMessage$(ActorCell.scala:353)
        at akka.actor.ActorCell.sendMessage(ActorCell.scala:433)
        at akka.actor.LocalActorRef.$bang(ActorRef.scala:400)
        at akka.stream.impl.io.TcpConnectionStage$TcpStreamLogic$$anon$6.onPush(TcpStages.scala:329)
        at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519)
        at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:411)
        at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:585)
        at akka.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute(ActorGraphInterpreter.scala:44)
        at akka.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute$(ActorGraphInterpreter.scala:40)
        at akka.stream.impl.fusing.ActorGraphInterpreter$BatchingActorInputBoundary$OnNext.execute(ActorGraphInterpreter.scala:77)
        at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:560)
        at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:742)
        at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:757)
        at akka.actor.Actor.aroundReceive(Actor.scala:517)
        at akka.actor.Actor.aroundReceive$(Actor.scala:515)
        at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:667)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:590)
        at akka.actor.ActorCell.invoke(ActorCell.scala:559)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-10-20 00:06:19,465 [INFO] from play.core.server.AkkaHttpServer in Thread-5 - Stopping server...
2018-10-20 00:10:19,104 [ERROR] from akka.actor.ActorSystemImpl in application-akka.actor.default-dispatcher-3 - Uncaught error from thread [application-akka.actor.default-dispatcher-5926]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]
java.lang.OutOfMemoryError: Java heap space
2018-10-20 00:11:23,112 [WARN] from org.elasticsearch.transport.TransportService in elasticsearch[_client_][transport_client_boss][T#10] - Received response for a request that has timed out, sent [30827ms] ago, timed out [12305ms] ago, action [cluster:monitor/nodes/liveness], node [{#transport#-1}{mTnQJOu6TSuJ8Oryp4q7jQ}{127.0.0.1}{127.0.0.1:9300}], id [257459]
2018-10-20 00:12:44,368 [INFO] from org.elasticsearch.client.transport.TransportClientNodesService in elasticsearch[_client_][generic][T#1] - failed to get node info for {#transport#-1}{mTnQJOu6TSuJ8Oryp4q7jQ}{127.0.0.1}{127.0.0.1:9300}, disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][127.0.0.1:9300][cluster:monitor/nodes/liveness] request_id [257459] timed out after [18522ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:961)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)