Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.06k forks source link

Output buffer is increasing and increasing #774

Closed timukas closed 9 years ago

timukas commented 9 years ago

Hi,

Just moved from 0.92.beta1 to 0.93-SNAP. Configuration is same in both setups:

output_batch_size = 1000 output_flush_interval = 1 processbuffer_processors = 20 outputbuffer_processors = 15 ring_size = 4096

But in 0.93-SNAP Output buffer is getting bigger and bigger - "807649 messages in output buffer. (19719.117% utilized) " Input buffer is around zero. Radio buffer is also around zero.

With 0.92.beta Output buffer were also around zero, but not with 0.93-SNAP version. My average throughput with both version is 4000 msg/sec.

upd. Buffer is increasing even if there's only one Gelf UDP input (nor radio inputs configured).

Any ideas, what configuration settings could be chenged/tuned?

regards,

timukas commented 9 years ago

And when go to Server nodes "details" pages on 0.93-SNAP there's a lot of errors "Unable to find node 546c9d14e4b0a6225730866a" in graylog2 server log file:

# [error] play - Cannot invoke the action, eventually got an error: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

[error] application -

! @6k92cd555 - Internal server error, for (GET) [/a/system/throughput/node/546c9d14e4b0a6225730866a] ->

play.api.Application$$anon$1: Execution exception[[RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
]]
        at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
Caused by: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:57) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at controllers.api.SystemApiController.nodeThroughput(SystemApiController.java:134) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:0.93.0-SNAPSHOT]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$132$$anonfun$apply$740.apply(routes_routing.scala:2318) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$132$$anonfun$apply$740.apply(routes_routing.scala:2318) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) ~[com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
Caused by: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

        at org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder.execute(ApiClientImpl.java:435) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:47) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at controllers.api.SystemApiController.nodeThroughput(SystemApiController.java:134) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:0.93.0-SNAPSHOT]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$132$$anonfun$apply$740.apply(routes_routing.scala:2318) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$132$$anonfun$apply$740.apply(routes_routing.scala:2318) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
[error] play - Cannot invoke the action, eventually got an error: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

[error] application -

! @6k92cd55b - Internal server error, for (GET) [/a/system/node/546c9d14e4b0a6225730866a/heap] ->

play.api.Application$$anon$1: Execution exception[[RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
]]
        at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
Caused by: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:57) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at controllers.api.SystemApiController.heap(SystemApiController.java:172) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:0.93.0-SNAPSHOT]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$137$$anonfun$apply$755.apply(routes_routing.scala:2358) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$137$$anonfun$apply$755.apply(routes_routing.scala:2358) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) ~[com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
Caused by: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

        at org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder.execute(ApiClientImpl.java:435) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:47) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at controllers.api.SystemApiController.heap(SystemApiController.java:172) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:0.93.0-SNAPSHOT]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$137$$anonfun$apply$755.apply(routes_routing.scala:2358) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$137$$anonfun$apply$755.apply(routes_routing.scala:2358) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
[error] play - Cannot invoke the action, eventually got an error: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

[error] play - Cannot invoke the action, eventually got an error: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

[error] application -

! @6k92cd6mo - Internal server error, for (GET) [/a/system/throughput/node/546c9d14e4b0a6225730866a] ->

play.api.Application$$anon$1: Execution exception[[RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
]]
        at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
Caused by: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:57) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at controllers.api.SystemApiController.nodeThroughput(SystemApiController.java:134) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:0.93.0-SNAPSHOT]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$132$$anonfun$apply$740.apply(routes_routing.scala:2318) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$132$$anonfun$apply$740.apply(routes_routing.scala:2318) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) ~[com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
Caused by: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

        at org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder.execute(ApiClientImpl.java:435) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:47) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at controllers.api.SystemApiController.nodeThroughput(SystemApiController.java:134) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:0.93.0-SNAPSHOT]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$132$$anonfun$apply$740.apply(routes_routing.scala:2318) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$132$$anonfun$apply$740.apply(routes_routing.scala:2318) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
[error] application -

! @6k92cd6mp - Internal server error, for (GET) [/a/system/node/546c9d14e4b0a6225730866a/heap] ->

play.api.Application$$anon$1: Execution exception[[RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
]]
        at play.api.Application$class.handleError(Application.scala:296) ~[com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.api.DefaultApplication.handleError(Application.scala:402) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
        at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.4.jar:na]
Caused by: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:57) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at controllers.api.SystemApiController.heap(SystemApiController.java:172) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:0.93.0-SNAPSHOT]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$137$$anonfun$apply$755.apply(routes_routing.scala:2358) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$137$$anonfun$apply$755.apply(routes_routing.scala:2358) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at play.core.Router$HandlerInvokerFactory$$anon$4.resultCall(Router.scala:264) ~[com.typesafe.play.play_2.10-2.3.6.jar:2.3.6]
Caused by: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:42900/system/cluster/nodes/546c9d14e4b0a6225730866a returned 500 Internal Server Error body: Unable to find node 546c9d14e4b0a6225730866a
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546c9d14e4b0a6225730866a
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

        at org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder.execute(ApiClientImpl.java:435) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at org.graylog2.restclient.models.NodeService.loadNode(NodeService.java:47) ~[org.graylog2.graylog2-rest-client-0.93.0-SNAPSHOT.jar:na]
        at controllers.api.SystemApiController.heap(SystemApiController.java:172) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:0.93.0-SNAPSHOT]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$137$$anonfun$apply$755.apply(routes_routing.scala:2358) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
        at Routes$$anonfun$routes$1$$anonfun$applyOrElse$137$$anonfun$apply$755.apply(routes_routing.scala:2358) ~[graylog2-web-interface.graylog2-web-interface-0.93.0-SNAPSHOT.jar:na]
timukas commented 9 years ago

Hi,

Made clean install of latest 0.93-SNAP (graylog2-server-0.93.0-SNAPSHOT-20141119184530 + graylog2-web-interface-0.93.0-SNAPSHOT-20141119144842). Empty mongo + empty ES + default GL configuration.

Started server + web interface without any inputs configured, this is initial configuration. Now i go to System->Nodes. I have only one server node there. And when i select Details from Action dropdown menu, graylog server writes bunch of error logs:

2014-11-20 10:24:28,618 ERROR: org.graylog2.plugin.rest.AnyExceptionClassMapper - Unhandled exception in REST resource
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2014-11-20 10:24:28,618 ERROR: org.graylog2.plugin.rest.AnyExceptionClassMapper - Unhandled exception in REST resource
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2014-11-20 10:24:29,701 ERROR: org.graylog2.plugin.rest.AnyExceptionClassMapper - Unhandled exception in REST resource
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2014-11-20 10:24:29,701 ERROR: org.graylog2.plugin.rest.AnyExceptionClassMapper - Unhandled exception in REST resource
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

As you can see, there're "Unable to find node 546da3e1e4b020d8bb624803" events. Mongo db has this node:

 {
   "_id": ObjectId("546da3e1e4b020d8bb624803"),
   "is_master": true,
   "transport_address": "http:\/\/192.168.x.xx:52xxx\/",
   "type": "SERVER",
   "last_seen": NumberInt(1416471945),
   "node_id": "aaaf9050-3d5f-4998-ac41-4a8a828020c3"
}   

Graylog webinterface also fires a lot of errors, when i access details page:

2014-11-20 10:30:54,931 - [ERROR] - from application in play-akka.actor.default-dispatcher-49

! @6k97mb84i - Internal server error, for (GET) [/a/system/throughput/node/546da3e1e4b020d8bb624803] ->

play.api.Application$$anon$1: Execution exception[[RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:52xxx/system/cluster/nodes/546da3e1e4b020d8bb624803 returned 500 Internal Server Error body: Unable to find node 546da3e1e4b020d8bb624803
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

Now, when i create GELF UDP input on localhost and start sending some events (approx 10-15 event/sec) then Outbuffer is getting bigger and bigger. Also i've noticed, that number of events beeing processed by node is not shown:

output_buffer

kroepke commented 9 years ago

Something runs very slowly, I assume the mapdb based disk cache doesn't cope well with large spools. We are currently reworking that completely for 0.93.

All of these errors indicate that there server just doesn't make any progress anymore. To the point where it doesn't check in to mongodb because it misses its own timeout. On Nov 20, 2014 9:38 AM, "timukas" notifications@github.com wrote:

Hi,

Made clean install of latest 0.93-SNAP (graylog2-server-0.93.0-SNAPSHOT-20141119184530 + graylog2-web-interface-0.93.0-SNAPSHOT-20141119144842). Empty mongo + empty ES + default GL configuration.

Started server + web interface without any inputs configured, this is initial configuration. Now i go to System->Nodes. I have only one server node there. And when i select Details from Action dropdown menu, graylog server writes bunch of error logs:

######################## 2014-11-20 10:24:28,618 ERROR: org.graylog2.plugin.rest.AnyExceptionClassMapper - Unhandled exception in REST resource org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803 at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81) at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030) at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2014-11-20 10:24:28,618 ERROR: org.graylog2.plugin.rest.AnyExceptionClassMapper - Unhandled exception in REST resource org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803 at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81) at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030) at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2014-11-20 10:24:29,701 ERROR: org.graylog2.plugin.rest.AnyExceptionClassMapper - Unhandled exception in REST resource org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803 at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81) at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030) at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2014-11-20 10:24:29,701 ERROR: org.graylog2.plugin.rest.AnyExceptionClassMapper - Unhandled exception in REST resource org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803 at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81) at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030) at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) ##########################

As you can see, there're "Unable to find node 546da3e1e4b020d8bb624803" events. Mongo db has this node: { "_id": ObjectId("546da3e1e4b020d8bb624803"), "is_master": true, "transport_address": "http:\/\/192.168.x.xx:52xxx\/", "type": "SERVER", "last_seen": NumberInt(1416471945), "node_id": "aaaf9050-3d5f-4998-ac41-4a8a828020c3" }

Graylog webinterface also fires a lot of errors, when i access details page:

############################ 2014-11-20 10:30:54,931 - [ERROR] - from application in play-akka.actor.default-dispatcher-49

! @6k97mb84i - Internal server error, for (GET) [/a/system/throughput/node/546da3e1e4b020d8bb624803] ->

play.api.Application$$anon$1: Execution exception[[RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.x.xx:52xxx/system/cluster/nodes/546da3e1e4b020d8bb624803 returned 500 Internal Server Error body: Unable to find node 546da3e1e4b020d8bb624803 org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803 at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81) at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030) at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) #################

Now, when i create GELF UDP input on localhost and start sending some events (approx 10-15 event/sec) then Outbuffer is getting bigger and bigger. Also i've noticed, that number of events beeing processed by node is not shown:

[image: output_buffer] https://cloud.githubusercontent.com/assets/1915618/5121812/546532f8-70a1-11e4-9299-0abed5841a8f.png

— Reply to this email directly or view it on GitHub https://github.com/Graylog2/graylog2-server/issues/774#issuecomment-63776168 .

timukas commented 9 years ago

I don't know will it help or no, but pressing on "Pause processing" under System->Nodes->xxxx also fires an errors "Unable to find node":

# [error] play - Cannot invoke the action, eventually got an error: java.lang.RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.4.55:52900/system/cluster/nodes/546da3e1e4b020d8bb624803 returned 500 Internal Server Error body: Unable to find node 546da3e1e4b020d8bb624803
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

[error] application -

! @6k998m2pd - Internal server error, for (GET) [/a/system/node/546da3e1e4b020d8bb624803/heap] ->

play.api.Application$$anon$1: Execution exception[[RuntimeException: org.graylog2.restclient.lib.APIException: API call failed GET http://@192.168.4.55:52900/system/cluster/nodes/546da3e1e4b020d8bb624803 returned 500 Internal Server Error body: Unable to find node 546da3e1e4b020d8bb624803
org.graylog2.cluster.NodeNotFoundException: Unable to find node 546da3e1e4b020d8bb624803
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:81)
        at org.graylog2.rest.resources.system.ClusterResource.node(ClusterResource.java:100)
        at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
        at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
        at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
        at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:622)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
timukas commented 9 years ago

Forgot to mention, this happens with graylog2-web-interface-0.93.0-SNAPSHOT-20141120115011 + graylog2-server-0.93.0-SNAPSHOT-20141120104108.

timukas commented 9 years ago

Here's some update on this issue. I'm sending same logs to two different graylog instances - 0.92-beta.1 and 0.93-SNAP.

Despite that i have huge output buffer on 0.93-SNAP version. I did not notice any missing events or some other anomalies. output_buffer_02

Number of events for same period are exactly the same on both graylog2 instances.

timukas commented 9 years ago

Hi,

Installed latest snapshot (graylog2-server-0.93.0-SNAPSHOT-20141127122812 + graylog2-radio-0.93.0-SNAPSHOT-20141127122812 + graylog2-web-interface-0.93.0-SNAPSHOT-20141127103658).

Now output buffer is not climbing up like crazy any more. Stays most of the time on 0. In very rare cases 1 or 2 messages in output buffer, but this imho is normal situation.

joschi commented 9 years ago

This has been a display problem (or rather a problem in calculating the utilization of the buffers) in the master and 0.92 branches which has been fixed by the above mentioned commits.