Open Srinathc opened 10 years ago
Full log:
INF [20140406-01:47:13.431] stats: Starting LatchedStatsListener INF [20140406-01:47:13.716] admin: Starting TimeSeriesCollector INF [20140406-01:47:13.719] admin: Admin HTTP interface started on port 2223. INF [20140406-01:47:13.732] kestrel: Kestrel config: listenAddress=0.0.0.0 memcachePort=Some(22133) textPort=Some(2222) queuePath=/var/spool/kestrel expirationTimerFrequency=Some(1.seconds) clientTimeout=None maxOpenTransactions=100 connectionBacklog=None statusFile=/tmp/.kestrel-status defaultStatus=Up statusChangeGracePeriod=0.seconds zookeeper=INF [20140406-01:47:14.030] kestrel: no status stored at '/tmp/.kestrel-status'; status remains 'Up' INF [20140406-01:47:15.019] kestrel: kestrel-memcache server started on /0.0.0.0:22133 INF [20140406-01:47:15.023] kestrel: kestrel-text server started on /0.0.0.0:2222 INF [20140406-01:47:15.103] kestrel: kestrel-thrift server started on /0.0.0.0:2229 INF [20140406-01:47:15.108] kestrel: Starting up background expiration task. INF [20140406-01:47:15.109] kestrel: Starting background-expiration INF [20140406-01:47:15.113] kestrel: Kestrel 2.4.1 started. INF [20140406-01:59:11.905] kestrel: Setting up queue PEP: maxItems=2147483647 maxSize=9223372036854775807.bytes maxItemSize=9223372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=134217728.bytes maxJournalSize=1073741824.bytes discardOldWhenFull=false keepJournal=true syncJournal=100.milliseconds expireToQueue=None maxExpireSweep=2147483647 fanoutOnly=false maxQueueAge=None (via ip-172-31-40-91.us-west-2.compute.internal:32901) INF [20140406-01:59:11.968] kestrel: Replaying transaction journal for 'PEP' INF [20140406-01:59:12.000] kestrel: No transaction journal for 'PEP'; starting with empty queue. INF [20140406-01:59:12.001] kestrel: Finished transaction journal for 'PEP' (0 items, 0 bytes) xid=0 INF [20140406-01:59:12.095] kestrel: Setting up queue PR: maxItems=2147483647 maxSize=9223372036854775807.bytes maxItemSize=9223372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=134217728.bytes maxJournalSize=1073741824.bytes discardOldWhenFull=false keepJournal=true syncJournal=100.milliseconds expireToQueue=None maxExpireSweep=2147483647 fanoutOnly=false maxQueueAge=None (via ip-172-31-40-91.us-west-2.compute.internal:32902) INF [20140406-01:59:12.099] kestrel: Replaying transaction journal for 'PR' INF [20140406-01:59:12.101] kestrel: No transaction journal for 'PR'; starting with empty queue. INF [20140406-01:59:12.101] kestrel: Finished transaction journal for 'PR' (0 items, 0 bytes) xid=0 INF [20140406-01:59:12.102] kestrel: Setting up queue SERVER: maxItems=2147483647 maxSize=9223372036854775807.bytes maxItemSize=9223372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=134217728.bytes maxJournalSize=1073741824.bytes discardOldWhenFull=false keepJournal=true syncJournal=100.milliseconds expireToQueue=None maxExpireSweep=2147483647 fanoutOnly=false maxQueueAge=None (via ip-172-31-40-91.us-west-2.compute.internal:32903) INF [20140406-01:59:12.104] kestrel: Replaying transaction journal for 'SERVER' INF [20140406-01:59:12.105] kestrel: No transaction journal for 'SERVER'; starting with empty queue. INF [20140406-01:59:12.106] kestrel: Finished transaction journal for 'SERVER' (0 items, 0 bytes) xid=0 INF [20140406-01:59:12.147] kestrel: Setting up queue SERVICE: maxItems=2147483647 maxSize=9223372036854775807.bytes maxItemSize=9223372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=134217728.bytes maxJournalSize=1073741824.bytes discardOldWhenFull=false keepJournal=true syncJournal=100.milliseconds expireToQueue=None maxExpireSweep=2147483647 fanoutOnly=false maxQueueAge=None (via ip-172-31-40-91.us-west-2.compute.internal:32906) INF [20140406-01:59:12.148] kestrel: Replaying transaction journal for 'SERVICE' INF [20140406-01:59:12.153] kestrel: No transaction journal for 'SERVICE'; starting with empty queue. INF [20140406-01:59:12.154] kestrel: Finished transaction journal for 'SERVICE' (0 items, 0 bytes) xid=0 FAT [20140407-14:24:57.950] kestrel-memcache: A server service threw an exception FAT [20140407-14:24:57.950] kestrel-memcache: java.lang.IndexOutOfBoundsException: 1 FAT [20140407-14:24:57.950] kestrel-memcache: at scala.collection.LinearSeqOptimized$class.apply(LinearSeqOptimized.scala:51) FAT [20140407-14:24:57.950] kestrel-memcache: at scala.collection.immutable.List.apply(List.scala:76) FAT [20140407-14:24:57.950] kestrel-memcache: at net.lag.kestrel.MemcacheHandler.handle(MemcacheHandler.scala:117) FAT [20140407-14:24:57.950] kestrel-memcache: at net.lag.kestrel.MemcacheHandler.apply(MemcacheHandler.scala:63) FAT [20140407-14:24:57.950] kestrel-memcache: at net.lag.kestrel.MemcacheHandler.apply(MemcacheHandler.scala:34) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.Service$$anon$1.apply(Service.scala:14) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:111) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.service.StatsFilter.apply(StatsFilter.scala:24) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:52) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.Filter$$anon$4.apply(Filter.scala:69) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.Service$$anon$1.apply(Service.scala:14) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.filter.MkJvmFilter$$anon$1.apply(JvmFilter.scala:19) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.tracing.TracingFilter$$anonfun$apply$1.apply(TracingFilter.scala:21) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.tracing.TracingFilter$$anonfun$apply$1.apply(TracingFilter.scala:16) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.tracing.Trace$.unwind(Trace.scala:128) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.tracing.TracingFilter.apply(TracingFilter.scala:16) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.filter.MonitorFilter$$anonfun$apply$1.apply(MonitorFilter.scala:16) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.finagle.filter.MonitorFilter$$anonfun$apply$1.apply(MonitorFilter.scala:16) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Future$$anonfun$monitored$1.apply$mcV$sp(Future.scala:128) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Monitor$$anonfun$apply$1.apply$mcV$sp(Monitor.scala:39) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:39) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:39) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Monitor$$anonfun$using$1.apply(Monitor.scala:104) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Monitor$.restoring(Monitor.scala:111) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Monitor$.using(Monitor.scala:102) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Monitor$class.apply(Monitor.scala:38) FAT [20140407-14:24:57.950] kestrel-memcache: at com.twitter.util.Monitor$$anon$1.apply(Monitor.scala:132) FAT [20140407-14:24:57.950] kestrel-memcache: (...more...)
While debugging an issue of why items from kestrel were not being consumed, it was found that kestrel initialization failed with the following exception.