search

graylog-labs / graylog2-web-interface

[DEPRECATED]
https://www.graylog.org/
611 stars 174 forks source link

Uncaught error from thread [play-akka.actor.default-dispatcher-33] #1547

Closed HSVK closed 9 years ago

HSVK commented 9 years ago

Hi, I am using the latest version of graylog 1.1.5, i happen to come across this issue today and the graylog-web service was down. Is this some kind of bug or configuration issue?

I was googling for akka, i saw that this line added to the configuration will eliminate the error, but not sure what caused this. 

akka.jvm-exit-on-fatal-error = off

Can you help please.

/var/log/graylog-web/console.log 

Play server process ID is 1511
Uncaught error from thread [play-akka.actor.default-dispatcher-33] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[play]
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672)
        at scala.concurrent.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1966)
        at scala.concurrent.forkjoin.ForkJoinPool.externalPush(ForkJoinPool.java:1829)
        at scala.concurrent.forkjoin.ForkJoinPool.execute(ForkJoinPool.java:2955)
        at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:120)
        at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
        at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
        at scala.concurrent.Promise$class.complete(Promise.scala:55)
        at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
        at scala.concurrent.Promise$class.success(Promise.scala:89)
        at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:153)
        at play.api.mvc.Filter$$anon$1$$anonfun$apply$4$$anonfun$apply$5.apply(Filters.scala:59)
        at play.api.mvc.Filter$$anon$1$$anonfun$apply$4$$anonfun$apply$5.apply(Filters.scala:57)
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483)
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[ERROR] [08/03/2015 15:26:54.629] [play-akka.actor.default-dispatcher-33] [ActorSystem(play)] Uncaught error from thread [play-akka.actor.default-dispatcher-33] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:714)
        at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672)
        at scala.concurrent.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1966)
        at scala.concurrent.forkjoin.ForkJoinPool.externalPush(ForkJoinPool.java:1829)
        at scala.concurrent.forkjoin.ForkJoinPool.execute(ForkJoinPool.java:2955)
        at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:120)
        at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
        at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
        at scala.concurrent.Promise$class.complete(Promise.scala:55)
        at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
        at scala.concurrent.Promise$class.success(Promise.scala:89)
        at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:153)
        at play.api.mvc.Filter$$anon$1$$anonfun$apply$4$$anonfun$apply$5.apply(Filters.scala:59)
        at play.api.mvc.Filter$$anon$1$$anonfun$apply$4$$anonfun$apply$5.apply(Filters.scala:57)
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483)
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496)
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

application.log: 

2015-08-03T15:14:54.436-06:00 - [ERROR] - from org.graylog2.restclient.lib.ApiClient in play-akka.actor.default-dispatcher-32
REST call failed
java.util.concurrent.TimeoutException: No response received after 5000
        at com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyResponseFuture.java:260) ~[com.ning.async-http-client-1.8.14.jar:na]
        at org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder.execute(ApiClientImpl.java:435) ~[org.graylog2.graylog2-rest-client--1.1.5-1.1.5.jar:na]
        at org.graylog2.restclient.models.dashboards.widgets.DashboardWidget.getValue(DashboardWidget.java:112) [org.graylog2.graylog2-rest-client--1.1.5-1.1.5.jar:na]
        at controllers.api.DashboardsApiController.widgetValue(DashboardsApiController.java:226) [graylog-web-interface.graylog-web-interface-1.1.5.jar:1.1.5]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$173$$anonfun$apply$957.apply(routes_routing.scala:2991) [graylog-web-interface.graylog-web-interface-1.1.5.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$173$$anonfun$apply$957.apply(routes_routing.scala:2991) [graylog-web-interface.graylog-web-interface-1.1.5.jar:na]
        at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.invocation(Router.scala:255) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:55) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.GlobalSettings$1.call(GlobalSettings.java:67) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.mvc.Security$AuthenticatedAction.call(Security.java:44) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.j.JavaAction$$anonfun$11.apply(JavaAction.scala:82) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:40) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Execution$trampoline$.execute(Execution.scala:46) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:32) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$.apply(Future.scala:31) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Future$.apply(Future.scala:485) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.core.j.JavaAction$class.apply(JavaAction.scala:82) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1.apply(Router.scala:252) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5.apply(Action.scala:130) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.utils.Threads$.withContextClassLoader(Threads.scala:21) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:129) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4.apply(Action.scala:128) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:128) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Action$$anonfun$apply$1.apply(Action.scala:121) [com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) [com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [org.scala-lang.scala-library-2.10.4.jar:na]
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.4.jar:na]

2015-08-03T15:26:54.674-06:00 - [ERROR] - from akka.actor.ActorSystemImpl in play-akka.actor.default-dispatcher-34
Uncaught error from thread [play-akka.actor.default-dispatcher-33] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) ~[na:1.7.0_85]
        at java.lang.Thread.start(Thread.java:714) ~[na:1.7.0_85]
        at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1966) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.externalPush(ForkJoinPool.java:1829) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.execute(ForkJoinPool.java:2955) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:120) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Promise$class.complete(Promise.scala:55) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.Promise$class.success(Promise.scala:89) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:153) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at play.api.mvc.Filter$$anon$1$$anonfun$apply$4$$anonfun$apply$5.apply(Filters.scala:59) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.mvc.Filter$$anon$1$$anonfun$apply$4$$anonfun$apply$5.apply(Filters.scala:57) ~[com.typesafe.play.play_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$mapM$1.apply(Iteratee.scala:483) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1.apply(Iteratee.scala:519) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14.apply(Iteratee.scala:496) ~[com.typesafe.play.play-iteratees_2.10-2.3.9.jar:2.3.9]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.10.4.jar:na]
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) ~[com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [com.typesafe.akka.akka-actor_2.10-2.3.5.jar:na]
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.4.jar:na]
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.4.jar:na]
joschi commented 9 years ago

I was googling for akka, i saw that this line added to the configuration will eliminate the error, but not sure what caused this.

akka.jvm-exit-on-fatal-error = off

Ignoring an error doesn't solve it. :wink:

java.lang.OutOfMemoryError: unable to create new native thread

The web interface ran out of memory which is a condition the JVM cannot recover from. As a workaround you should assign more heap memory to the web interface (see https://www.playframework.com/documentation/2.3.x/ProductionConfiguration#Specifying-additional-JVM-arguments for details).

Unfortunately the stack traces you've posted don't hint to the actual source of the problem but only contain technical classes of the framework.

Please use our mailing list or our IRC channel #graylog on Freenode for general questions about Graylog in the future.