search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.36k stars 1.06k forks source link

Graylog2-server 0.90.3 keeps crashing #925

Closed neilferreira closed 9 years ago

neilferreira commented 9 years ago

Hi all

My Graylog2 instance keeps running out of memory according to the logs, this is a daily occurrence where the instance would crash.

Is this the Graylog2 instance running out of memory, or Elasticsearch running out of memory?

SendRequestTransportException[[Pretty Persuasions][inet[/10.243.11.143:9300]][bulk/shard]]; nested: OutOfMemoryError[Java heap space];

2015-01-24 09:14:29,397 WARN : org.jboss.netty.channel.socket.nio.AbstractNioSelector - Unexpected exception in the selector loop. 2015-01-24 09:14:42,263 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering. Exception in thread "restapi-boss-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "elasticsearch[graylog2-server][transport_client_worker][T#1]" java.lang.OutOfMemoryError: Java heap space 2015-01-24 09:15:27,803 WARN : org.jboss.netty.channel.socket.nio.AbstractNioSelector - Failed to accept a connection. 2015-01-24 09:15:30,489 WARN : org.jboss.netty.channel.socket.nio.AbstractNioSelector - Unexpected exception in the selector loop. 2015-01-24 09:15:35,659 WARN : org.elasticsearch.netty.channel.socket.nio.AbstractNioSelector - Unexpected exception in the selector loop. 2015-01-24 09:16:50,943 WARN : org.graylog2.streams.StreamRouter - Processing of stream <539fbf32e4b0921410c2f509> failed to return within 2000ms. 2015-01-24 09:17:07,066 WARN : org.graylog2.streams.StreamRouter - Processing of stream <539fbf32e4b0921410c2f509> failed to return within 2000ms. 2015-01-24 09:17:21,854 WARN : org.graylog2.streams.StreamRouter - Processing of stream <53996e47e4b0d6c5cf8beba7> failed to return within 2000ms. 2015-01-24 09:18:41,555 WARN : org.graylog2.streams.StreamRouter - Processing of stream <539fbf32e4b0921410c2f509> failed to return within 2000ms. Exception in thread "cluster-1-127.0.0.1:27017" java.lang.OutOfMemoryError: Java heap space Exception in thread "elasticsearch[graylog2-server][transport_client_boss][T#1]{New I/O boss #3}" java.lang.OutOfMemoryError: Java heap space 2015-01-24 09:19:22,121 WARN : org.graylog2.streams.StreamRouter - Processing of stream <539fbf32e4b0921410c2f509> failed to return within 2000ms. 2015-01-24 09:19:23,521 WARN : org.graylog2.streams.StreamRouter - Processing of stream <539fbf32e4b0921410c2f509> failed to return within 2000ms. 2015-01-24 09:19:28,560 WARN : org.graylog2.streams.StreamRouter - Processing of stream <53996e47e4b0d6c5cf8beba7> failed to return within 2000ms. 2015-01-24 09:19:29,630 WARN : org.graylog2.streams.StreamRouter - Processing of stream <53996e47e4b0d6c5cf8beba7> failed to return within 2000ms. 2015-01-24 09:19:30,421 WARN : org.graylog2.streams.StreamRouter - Processing of stream <53996e47e4b0d6c5cf8beba7> failed to return within 2000ms. 2015-01-24 09:19:31,809 WARN : org.graylog2.streams.StreamRouter - Processing of stream <53996e47e4b0d6c5cf8beba7> failed to return within 2000ms. 2015-01-24 09:20:31,155 WARN : org.graylog2.buffers.processors.OutputBufferProcessor - Timeout reached. Not waiting any longer for writer threads to complete. Exception in thread "processbufferprocessor-1" java.lang.OutOfMemoryError: Java heap space Exception in thread "processbufferprocessor-3" java.lang.OutOfMemoryError: Java heap space Exception in thread "processbufferprocessor-4" java.lang.OutOfMemoryError: Java heap space Exception in thread "processbufferprocessor-2" java.lang.OutOfMemoryError: Java heap space Exception in thread "processbufferprocessor-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "input-53981869e4b015383530313c-gelfudp-worker-0" java.lang.OutOfMemoryError: Java heap space 2015-01-24 09:24:35,797 ERROR: com.lmax.disruptor.FatalExceptionHandler - Exception processing: 228776 org.graylog2.plugin.buffers.MessageEvent@3ad386b4 2015-01-24 09:24:44,848 ERROR: com.lmax.disruptor.FatalExceptionHandler - Exception processing: 228778 org.graylog2.plugin.buffers.MessageEvent@4b42ee7b 2015-01-24 09:24:49,507 ERROR: com.lmax.disruptor.FatalExceptionHandler - Exception processing: 228779 org.graylog2.plugin.buffers.MessageEvent@dc9921f 2015-01-24 09:24:52,063 ERROR: com.lmax.disruptor.FatalExceptionHandler - Exception processing: 228777 org.graylog2.plugin.buffers.MessageEvent@6164c36c 2015-01-24 09:24:52,071 ERROR: com.lmax.disruptor.FatalExceptionHandler - Exception processing: 228775 org.graylog2.plugin.buffers.MessageEvent@1e5a1968 2015-01-24 09:38:46,846 WARN : org.jboss.netty.channel.socket.nio.AbstractNioSelector - Unexpected exception in the selector loop. 2015-01-24 09:57:58,939 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...

lennartkoopmann commented 9 years ago

This is your graylog2-server running out of heap space. How much heap space did you allocate in the JVM (-Xmx startup parameter), how many messages are you processing per second and how much RAM does the machine have in total?

neilferreira commented 9 years ago

I've increased it to 1gb, the machine has 4gb in total.

I'm trying to track down the messages it was getting at the time of going down, then getting this: ! @6l270c2ip - Internal server error, for (GET) [/search?rangetype=relative&width=2495&relative=1209600&from=&to=&q=&page=303&fields=source] ->

play.api.Application$$anon$1: Execution exception[[RuntimeException: java.lang.OutOfMemoryError: Java heap space]] at play.api.Application$class.handleError(Application.scala:293) ~[com.typesafe.play.play_2.10-2.2.2.jar:2.2.2] at play.api.DefaultApplication.handleError(Application.scala:399) [com.typesafe.play.play_2.10-2.2.2.jar:2.2.2] at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$3.apply(PlayDefaultUpstreamHandler.scala:261) [com.typesafe.play.play_2.10-2.2.2.jar:2.2.2] at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2$$anonfun$applyOrElse$3.apply(PlayDefaultUpstreamHandler.scala:261) [com.typesafe.play.play_2.10-2.2.2.jar:2.2.2] at scala.Option.map(Option.scala:145) [org.scala-lang.scala-library-2.10.3.jar:na] at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2.applyOrElse(PlayDefaultUpstreamHandler.scala:261) [com.typesafe.play.play_2.10-2.2.2.jar:2.2.2] at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$2.applyOrElse(PlayDefaultUpstreamHandler.scala:257) [com.typesafe.play.play_2.10-2.2.2.jar:2.2.2] at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:344) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:343) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.3.jar:na] at play.api.libs.iteratee.Execution$$anon$1.execute(Execution.scala:43) [com.typesafe.play.play-iteratees_2.10-2.2.2.jar:2.2.2] at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.Promise$class.complete(Promise.scala:55) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.forkjoin.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1361) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.10.3.jar:na] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.10.3.jar:na] Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space

That is on page 303 of the web interface, all the pages around it load just fine? Any other way of finding out what messages are in the DB on that page?

pkoraca commented 9 years ago

I have similar problem on v0.92.4. Processing up to 20k logs per second on 3 instances. Graylog2-server went OOM overnight while there was ~5k logs per second.

Each instance has:

Unforunately I lost my logs after graylog2-server restart.

However, I do see some MasterCache exceptions that may be related to OOM:

09:00:15,600 ERROR [AbstractCacheWorkerThread] Exception while working on cache: java.lang.NullPointerException at org.mapdb.Queues$SimpleQueue.poll(Queues.java:135) at org.mapdb.Queues$SimpleQueue.drainTo(Queues.java:271) at org.graylog2.caches.DiskJournalCache.drainTo(DiskJournalCache.java:210) at org.graylog2.periodical.AbstractCacheWorkerThread.drainMessagesEnqueue(AbstractCacheWorkerThread.java:79) at org.graylog2.periodical.AbstractCacheWorkerThread.work(AbstractCacheWorkerThread.java:58) at org.graylog2.periodical.InputCacheWorkerThread$1.run(InputCacheWorkerThread.java:60) at java.lang.Thread.run(Thread.java:745) 09:00:15,600 ERROR [MasterCacheWorkerThread] Error while trying to work on MasterCache . java.lang.NullPointerException

joschi commented 9 years ago

Please try to run Graylog with the JVM options we're using in the graylogctl command (without java.library.path): https://github.com/Graylog2/graylog2-server/blob/1.0.0-beta.3/bin/graylogctl#L54

neilferreira commented 9 years ago

Went down again after the weekend.

2015-01-29 22:53:06,405 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-29 23:04:03,698 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-29 23:04:03,760 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-29 23:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-29 23:06:58,639 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-29 23:11:19,951 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-29 23:11:20,047 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-29 23:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-29 23:42:51,570 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-29 23:53:06,358 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-29 23:53:06,489 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 00:04:03,741 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 00:04:03,795 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 00:06:58,489 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 00:06:58,583 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 00:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 00:11:20,066 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 00:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 00:42:51,363 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 00:53:06,328 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 00:53:06,527 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 01:00:12,177 WARN : org.graylog2.streams.StreamRouter - Processing of stream <539fbf32e4b0921410c2f509> failed to return within 2000ms.
2015-01-30 01:04:03,698 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 01:04:03,733 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 01:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 01:06:58,725 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 01:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 01:11:19,907 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 01:23:35,431 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Enabling session validation scheduler...
2015-01-30 01:42:51,325 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 01:42:51,410 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 01:53:06,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 01:53:06,506 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 02:04:03,698 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 02:04:03,779 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 02:06:58,459 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 02:06:58,524 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 02:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 02:11:19,911 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 02:23:35,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 02:23:35,506 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 02:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 02:42:51,355 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 02:53:06,328 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 02:53:07,626 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 03:04:03,705 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 03:04:03,750 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 03:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 03:06:58,656 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 03:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 03:11:19,955 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 03:23:35,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 03:23:35,643 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 03:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 03:42:51,389 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 03:53:06,328 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 03:53:06,550 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 04:04:03,698 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 04:04:03,721 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 04:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 04:06:58,647 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 04:11:20,011 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 04:11:20,098 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 04:23:35,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 04:23:35,647 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 04:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 04:42:52,395 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 04:53:06,328 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 04:53:06,554 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 05:04:03,733 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 05:04:03,843 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 05:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 05:06:58,645 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 05:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 05:11:19,907 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 05:23:35,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 05:23:35,628 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 05:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 05:42:51,564 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 05:53:06,819 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 05:53:06,866 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 06:04:03,736 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 06:04:03,830 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 06:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 06:06:58,489 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 06:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 06:11:20,072 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 06:23:35,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 06:23:35,464 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 06:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 06:42:51,505 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 06:53:06,407 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 06:53:06,488 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 07:04:03,698 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 07:04:03,891 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 07:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 07:06:58,479 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 07:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 07:11:20,047 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 07:23:35,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 07:23:35,460 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 07:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 07:42:51,527 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 07:53:06,460 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 07:53:06,562 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 08:04:03,698 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 08:04:03,911 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 08:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 08:06:59,519 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 08:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 08:11:20,065 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 08:23:35,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 08:23:35,464 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 08:42:51,613 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 08:42:51,901 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 08:53:06,328 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 08:53:06,520 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 09:04:03,698 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 09:04:03,879 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 09:06:58,577 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 09:06:58,682 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 09:11:19,872 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 09:11:20,995 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 09:23:35,433 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 09:23:35,472 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 09:42:51,324 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 09:42:51,421 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  [1] sessions were stopped.
2015-01-30 09:47:58,979 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
    at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:142)
    at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
    at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
    at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
    at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
    at java.io.BufferedWriter.flush(BufferedWriter.java:254)
    at org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
    at org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 09:48:23,233 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
    at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:142)
    at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
    at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
    at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
    at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
    at java.io.BufferedWriter.flush(BufferedWriter.java:254)
    at org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
    at org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 09:48:30,746 WARN : org.elasticsearch.transport - [graylog2-server] Transport response handler not found of id [1720785]
2015-01-30 09:48:29,495 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
    at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:142)
    at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
    at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
    at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
    at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
    at java.io.BufferedWriter.flush(BufferedWriter.java:254)
    at org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
    at org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Exception in thread "restapi-boss-0" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "restapi-boss-0"
Exception in thread "elasticsearch[graylog2-server][transport_client_worker][T#1]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "input-53981869e4b015383530313c-gelfudp-worker-0" java.lang.NullPointerException

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "input-53981869e4b015383530313c-gelfudp-worker-0"
2015-01-30 09:48:53,511 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:49:07,772 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:49:25,100 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x79664e37, /127.0.0.1:56727 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:49:27,043 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x79664e37, /127.0.0.1:56727 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:49:28,855 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x79664e37, /127.0.0.1:56727 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:49:30,814 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x4f3a61ac, /127.0.0.1:56706 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:49:33,845 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x4f3a61ac, /127.0.0.1:56706 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:53:51,142 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x4f3a61ac, /127.0.0.1:56706 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:53:53,894 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:53:56,695 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x4f3a61ac, /127.0.0.1:56706 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:49:35,192 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:54:04,212 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x79664e37, /127.0.0.1:56727 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:54:00,390 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x4f3a61ac, /127.0.0.1:56706 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:54:45,465 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 09:54:40,183 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 09:54:07,967 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x79664e37, /127.0.0.1:56727 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space
2015-01-30 09:55:44,197 ERROR: com.lmax.disruptor.FatalExceptionHandler - Exception processing: 415139 org.graylog2.plugin.buffers.MessageEvent@619766e8
com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
    at org.graylog2.streams.CachedStreamRouter.getStreams(CachedStreamRouter.java:94)
    at org.graylog2.streams.StreamRouter.route(StreamRouter.java:98)
    at org.graylog2.filters.StreamMatcherFilter.filter(StreamMatcherFilter.java:50)
    at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:96)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:41)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: Java heap space
Exception in thread "processbufferprocessor-4" java.lang.RuntimeException: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:152)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
    at org.graylog2.streams.CachedStreamRouter.getStreams(CachedStreamRouter.java:94)
    at org.graylog2.streams.StreamRouter.route(StreamRouter.java:98)
    at org.graylog2.filters.StreamMatcherFilter.filter(StreamMatcherFilter.java:50)
    at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:96)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:41)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
    ... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2015-01-30 09:55:41,471 ERROR: com.lmax.disruptor.FatalExceptionHandler - Exception processing: 415140 org.graylog2.plugin.buffers.MessageEvent@45e48f4a
com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
    at org.graylog2.streams.CachedStreamRouter.getStreams(CachedStreamRouter.java:94)
    at org.graylog2.streams.StreamRouter.route(StreamRouter.java:98)
    at org.graylog2.filters.StreamMatcherFilter.filter(StreamMatcherFilter.java:50)
    at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:96)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:41)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: Java heap space
Exception in thread "processbufferprocessor-0" java.lang.RuntimeException: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:152)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
    at org.graylog2.streams.CachedStreamRouter.getStreams(CachedStreamRouter.java:94)
    at org.graylog2.streams.StreamRouter.route(StreamRouter.java:98)
    at org.graylog2.filters.StreamMatcherFilter.filter(StreamMatcherFilter.java:50)
    at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:96)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:41)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
    ... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2015-01-30 09:55:14,132 ERROR: com.lmax.disruptor.FatalExceptionHandler - Exception processing: 415138 org.graylog2.plugin.buffers.MessageEvent@37fd037
com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
    at org.graylog2.streams.CachedStreamRouter.getStreams(CachedStreamRouter.java:94)
    at org.graylog2.streams.StreamRouter.route(StreamRouter.java:98)
    at org.graylog2.filters.StreamMatcherFilter.filter(StreamMatcherFilter.java:50)
    at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:96)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:41)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: Java heap space
Exception in thread "processbufferprocessor-3" java.lang.RuntimeException: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:152)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
    at org.graylog2.streams.CachedStreamRouter.getStreams(CachedStreamRouter.java:94)
    at org.graylog2.streams.StreamRouter.route(StreamRouter.java:98)
    at org.graylog2.filters.StreamMatcherFilter.filter(StreamMatcherFilter.java:50)
    at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:96)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86)
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:41)
    at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
    ... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2015-01-30 09:54:53,032 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x4f3a61ac, /127.0.0.1:56706 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space
2015-01-30 09:56:12,791 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
    at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter.commit(NettyContainer.java:198)
    at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:412)
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:691)
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 09:55:54,741 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
    at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter.commit(NettyContainer.java:198)
    at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:412)
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:691)
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 09:56:35,379 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:56:39,170 INFO : org.elasticsearch.discovery.zen - [graylog2-server] master_left [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], reason [failed to ping, tried [3] times, each with  maximum [30s] timeout]
2015-01-30 09:56:59,686 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:57:03,397 WARN : org.elasticsearch.discovery.zen - [graylog2-server] master_left and no other node elected to become master, current nodes: {[graylog2-server][dS-5gL-rT4OiPUNh3vIufA][inet[/10.243.11.143:9350]]{client=true, data=false, master=false},}
2015-01-30 09:57:03,396 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x4fc030c4, /127.0.0.1:56768 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
java.lang.OutOfMemoryError: Java heap space
2015-01-30 09:57:40,137 INFO : org.elasticsearch.cluster.service - [graylog2-server] removed {[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]],}, reason: zen-disco-master_failed ([Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]])
2015-01-30 09:57:44,504 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x25eacf67, /127.0.0.1:56831 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
java.lang.OutOfMemoryError: Java heap space
2015-01-30 09:58:50,659 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:59:00,658 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xd6f325ad, /127.0.0.1:56873 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:59:03,441 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xd6f325ad, /127.0.0.1:56873 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:59:16,911 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:59:26,906 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:59:28,711 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:59:30,531 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
2015-01-30 09:59:32,348 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:59:34,161 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:59:35,978 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:59:37,779 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:59:39,597 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:59:41,429 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:59:43,246 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:59:45,065 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
2015-01-30 09:59:46,880 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:59:48,695 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space
2015-01-30 09:59:53,242 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 09:59:48,696 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xd6f325ad, /127.0.0.1:56873 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
java.lang.OutOfMemoryError: Java heap space
2015-01-30 10:00:04,977 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0xb6125af5, /127.0.0.1:56810 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
java.lang.OutOfMemoryError: Java heap space
2015-01-30 10:00:04,975 WARN : org.elasticsearch.transport.netty - [graylog2-server] exception caught on transport layer [[id: 0x6a06df24, /10.243.11.143:48149 => /10.243.11.143:9350]], closing connection
java.lang.OutOfMemoryError: Java heap space
2015-01-30 10:00:04,975 WARN : org.jboss.netty.util.HashedWheelTimer - An exception was thrown by TimerTask.
java.lang.OutOfMemoryError: Java heap space
2015-01-30 10:00:25,472 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
2015-01-30 10:00:37,154 WARN : org.jboss.netty.channel.DefaultChannelPipeline - An exception was thrown by a user handler while handling an exception event ([id: 0x0df998a0, /127.0.0.1:56852 => /127.0.0.1:12900] EXCEPTION: java.io.IOException: Broken pipe)
java.lang.OutOfMemoryError: Java heap space
2015-01-30 10:00:38,221 WARN : org.elasticsearch.discovery.zen.ping.multicast - [graylog2-server] received ping response ping_response{target [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], cluster_name[graylog2]} with no matching id [2]
2015-01-30 10:00:40,669 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
    at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:142)
    at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
    at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
    at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
    at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
    at java.io.BufferedWriter.flush(BufferedWriter.java:254)
    at org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
    at org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:00:52,945 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:00:58,393 ERROR: org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception during jersey resource handling
java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:51)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
    at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:142)
    at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:299)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
    at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
    at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
    at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
    at java.io.BufferedWriter.flush(BufferedWriter.java:254)
    at org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
    at org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
    at org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:265)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:106)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
    at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154)
    at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:621)
    at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:377)
    at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:367)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1028)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:01:07,636 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:01:11,793 ERROR: org.graylog2.periodical.IndexRetentionThread - Uncaught exception in periodical
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
    at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:89)
    at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:53)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.<init>(TransportBroadcastOperationAction.java:124)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:75)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:49)
    at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:65)
    at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:405)
    at org.graylog2.indexer.indices.Indices.getAll(Indices.java:152)
    at org.graylog2.indexer.Deflector.getAllDeflectorIndices(Deflector.java:216)
    at org.graylog2.periodical.IndexRetentionThread.doRun(IndexRetentionThread.java:72)
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:86)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:01:57,883 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:02:23,396 ERROR: org.graylog2.periodical.IndexRetentionThread - Uncaught exception in periodical
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
    at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:89)
    at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:53)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.<init>(TransportBroadcastOperationAction.java:124)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:75)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:49)
    at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:65)
    at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:405)
    at org.graylog2.indexer.indices.Indices.getAll(Indices.java:152)
    at org.graylog2.indexer.Deflector.getAllDeflectorIndices(Deflector.java:216)
    at org.graylog2.periodical.IndexRetentionThread.doRun(IndexRetentionThread.java:72)
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:86)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:02:50,727 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:03:42,396 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:04:04,413 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 10:04:32,431 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 10:04:33,495 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:04:34,724 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:05:26,724 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:06:19,041 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:06:59,166 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 10:07:44,182 ERROR: org.graylog2.periodical.IndexRetentionThread - Uncaught exception in periodical
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];
    at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:138)
    at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:89)
    at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:53)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.<init>(TransportBroadcastOperationAction.java:124)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:75)
    at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:46)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:49)
    at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:65)
    at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:405)
    at org.graylog2.indexer.indices.Indices.getAll(Indices.java:152)
    at org.graylog2.indexer.Deflector.getAllDeflectorIndices(Deflector.java:216)
    at org.graylog2.periodical.IndexRetentionThread.doRun(IndexRetentionThread.java:72)
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:86)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:08:45,307 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation.  No sessions were stopped.
2015-01-30 10:08:46,351 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:08:47,565 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:09:37,657 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:10:29,262 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:11:19,595 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:11:20,816 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 10:12:29,006 ERROR: org.graylog2.periodical.IndexRetentionThread - Uncaught exception in periodical
2015-01-30 10:14:07,825 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:14:54,036 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:15:46,516 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:16:40,159 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:16:42,471 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:17:23,412 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:17:33,486 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:18:18,313 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:18:30,925 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:19:14,609 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:19:27,104 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:19:40,704 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:20:13,573 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:20:29,454 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:20:31,755 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:20:45,395 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:20:51,307 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:20:59,346 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:21:05,815 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:21:11,446 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:21:15,957 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:21:21,387 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:21:29,431 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:21:32,770 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:21:35,069 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
org.elasticsearch.transport.ConnectTransportException: [Pretty Persuasions][inet[/10.243.11.143:9300]] connect_timeout[30s]
    at org.elasticsearch.transport.netty.NettyTransport.connectToChannels(NettyTransport.java:711)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:640)
    at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:608)
    at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:129)
    at org.elasticsearch.discovery.zen.ZenDiscovery.innerJoinCluster(ZenDiscovery.java:335)
    at org.elasticsearch.discovery.zen.ZenDiscovery.access$500(ZenDiscovery.java:76)
    at org.elasticsearch.discovery.zen.ZenDiscovery$1.run(ZenDiscovery.java:283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:21:40,401 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:21:49,472 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:22:06,793 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:22:31,382 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:22:41,269 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:22:52,738 WARN : org.elasticsearch.discovery.zen - [graylog2-server] failed to connect to master [[Pretty Persuasions][l0eiK7PmTB-THVoLo77eng][inet[/10.243.11.143:9300]]], retrying...
2015-01-30 10:23:00,434 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
Exception in thread "elasticsearch[graylog2-server][generic][T#55]" java.lang.OutOfMemoryError: Java heap space
2015-01-30 10:23:05,981 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:23:10,537 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:23:14,037 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:23:18,241 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:23:21,735 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:23:27,200 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:23:35,006 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:23:37,274 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 10:23:51,967 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:23:55,740 ERROR: org.graylog2.periodical.IndexerClusterCheckerThread - Uncaught exception in periodical
java.lang.IllegalArgumentException: response too long: 1714631680
    at com.mongodb.Response.<init>(Response.java:49)
    at com.mongodb.DBPort$1.execute(DBPort.java:141)
    at com.mongodb.DBPort$1.execute(DBPort.java:135)
    at com.mongodb.DBPort.doOperation(DBPort.java:164)
    at com.mongodb.DBPort.call(DBPort.java:135)
    at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:292)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:271)
    at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84)
    at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
    at com.mongodb.DBCollection.findOne(DBCollection.java:870)
    at com.mongodb.DBCollection.findOne(DBCollection.java:844)
    at com.mongodb.DBCollection.findOne(DBCollection.java:790)
    at org.graylog2.database.PersistedServiceImpl.findOne(PersistedServiceImpl.java:133)
    at org.graylog2.notifications.NotificationServiceImpl.isFirst(NotificationServiceImpl.java:71)
    at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:45)
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:86)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
2015-01-30 10:23:58,600 WARN : org.graylog2.periodical.NodePingThread - Did not find meta info of this node. Re-registering.
2015-01-30 10:42:51,319 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 11:04:03,958 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
2015-01-30 11:06:58,452 INFO : org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions...
bernd commented 9 years ago

Can you please test this with Graylog 1.0.0 and re-open the issue if you still see this problem? There have been lots of improvements regarding performance and stability in 1.0.0.

Thank you!

andreaconsadori commented 9 years ago

maybe the issue is the same, on graylog 1.0.0

Caused by: com.mongodb.MongoTimeoutException: Timed out after 10000 ms while waiting to connect. Client view of cluster state is {type=Unknown, servers=[{address=127.0.0.1:27017, type=Unknown, state=Connecting, exception={java.lang.OutOfMemoryError: Java heap space}}]
        at com.mongodb.BaseCluster.getDescription(BaseCluster.java:128)
        at com.mongodb.DBTCPConnector.getClusterDescription(DBTCPConnector.java:396)
        at com.mongodb.DBTCPConnector.getMaxBsonObjectSize(DBTCPConnector.java:641)
        at com.mongodb.Mongo.getMaxBsonObjectSize(Mongo.java:641)
        at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:81)
        at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66)
        at com.mongodb.DBCollection.findOne(DBCollection.java:869)
        at com.mongodb.DBCollection.findOne(DBCollection.java:843)
        at com.mongodb.DBCollection.findOne(DBCollection.java:789)
        at org.graylog2.database.PersistedServiceImpl.get(PersistedServiceImpl.java:55)
        at org.graylog2.database.PersistedServiceImpl.get(PersistedServiceImpl.java:59)
        at org.graylog2.inputs.InputServiceImpl.find(InputServiceImpl.java:113)
        at org.graylog2.filters.StaticFieldFilter$1.call(StaticFieldFilter.java:78)
        at org.graylog2.filters.StaticFieldFilter$1.call(StaticFieldFilter.java:74)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        ... 12 more
joschi commented 9 years ago

@andreaconsadori Please make sure that you're using the JVM options provided by our startup scripts to run Graylog (https://github.com/Graylog2/graylog2-server/blob/1.0.0/bin/graylogctl#L54).

andreaconsadori commented 9 years ago

it's a debian package install and under /etc/defaul/graylog-server i put GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -XX:PermSize=512m -XX:MaxPermSize=1024m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow"

but issue still persist after reboot

Exception in thread "syslog-tcp-transport-worker-4" java.lang.OutOfMemoryError: Java heap space Exception in thread "elasticsearch[graylog2-server][generic][T#191]" java.lang.OutOfMemoryError: Java heap space Exception in thread "pool-26-thread-1" Exception in thread "processbufferprocessor-4" java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread "elasticsearch[graylog2-server][discovery#multicast#receiver][T#1]" java.lang.OutOfMemoryError: Java heap space Exception in thread "JournalReader" java.lang.OutOfMemoryError: Java heap space

andreaconsadori commented 9 years ago

after remming out in vim /etc/default/elasticsearch

ES_HEAP_SIZE=2g

the issue disappear

bernd commented 9 years ago

Is there a reason you changed the default PermSize settings to these huge values? (default is -XX:PermSize=128m -XX:MaxPermSize=256m)

You should be able to run fine with the default settings.

andreaconsadori commented 9 years ago

inexperience :)

andreaconsadori commented 9 years ago

memoryusage

it's normal as memory usage?

if i revert PermSize and MaxPermSize Graylog cash, now with -XX:PermSize=256m -XX:MaxPermSize=512m is fine