search

twitter-archive / kestrel

simple, distributed message queue system (inactive)
http://twitter.github.io/kestrel
Other
2.78k stars 313 forks source link

Freshly installed kestrel server fails with java.lang.IndexOutOfBoundsException #141

Open Srinathc opened 10 years ago

Srinathc commented 10 years ago

While debugging an issue of why items from kestrel were not being consumed, it was found that kestrel initialization failed with the following exception.

FAT [20140407-14:24:57.950] kestrel-memcache: A server service  threw an exception
FAT [20140407-14:24:57.950] kestrel-memcache: java.lang.IndexOutOfBoundsException: 1
FAT [20140407-14:24:57.950] kestrel-memcache:     at scala.collection.LinearSeqOptimized$class.apply(LinearSeqOptimized.scala:51)
FAT [20140407-14:24:57.950] kestrel-memcache:     at scala.collection.immutable.List.apply(List.scala:76)
FAT [20140407-14:24:57.950] kestrel-memcache:     at net.lag.kestrel.MemcacheHandler.handle(MemcacheHandler.scala:117)
FAT [20140407-14:24:57.950] kestrel-memcache:     at net.lag.kestrel.MemcacheHandler.apply(MemcacheHandler.scala:63)
FAT [20140407-14:24:57.950] kestrel-memcache:     at net.lag.kestrel.MemcacheHandler.apply(MemcacheHandler.scala:34)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:111)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.service.StatsFilter.apply(StatsFilter.scala:24)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:52)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$4.apply(Filter.scala:69)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.filter.MkJvmFilter$$anon$1.apply(JvmFilter.scala:19)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.tracing.TracingFilter$$anonfun$apply$1.apply(TracingFilter.scala:21)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.tracing.TracingFilter$$anonfun$apply$1.apply(TracingFilter.scala:16)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.tracing.Trace$.unwind(Trace.scala:128)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.tracing.TracingFilter.apply(TracingFilter.scala:16)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.filter.MonitorFilter$$anonfun$apply$1.apply(MonitorFilter.scala:16)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.filter.MonitorFilter$$anonfun$apply$1.apply(MonitorFilter.scala:16)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Future$$anonfun$monitored$1.apply$mcV$sp(Future.scala:128)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anonfun$apply$1.apply$mcV$sp(Monitor.scala:39)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:39)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:39)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anonfun$using$1.apply(Monitor.scala:104)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$.restoring(Monitor.scala:111)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$.using(Monitor.scala:102)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$class.apply(Monitor.scala:38)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anon$1.apply(Monitor.scala:132)
FAT [20140407-14:24:57.950] kestrel-memcache:     (...more...)
Srinathc commented 10 years ago

Full log:

INF [20140406-01:47:13.431] stats: Starting LatchedStatsListener
INF [20140406-01:47:13.716] admin: Starting TimeSeriesCollector
INF [20140406-01:47:13.719] admin: Admin HTTP interface started on port 2223.
INF [20140406-01:47:13.732] kestrel: Kestrel config: listenAddress=0.0.0.0 memcachePort=Some(22133) textPort=Some(2222) queuePath=/var/spool/kestrel expirationTimerFrequency=Some(1.seconds) clientTimeout=None maxOpenTransactions=100 connectionBacklog=None statusFile=/tmp/.kestrel-status defaultStatus=Up statusChangeGracePeriod=0.seconds zookeeper=
INF [20140406-01:47:14.030] kestrel: no status stored at '/tmp/.kestrel-status'; status remains 'Up'
INF [20140406-01:47:15.019] kestrel: kestrel-memcache server started on /0.0.0.0:22133
INF [20140406-01:47:15.023] kestrel: kestrel-text server started on /0.0.0.0:2222
INF [20140406-01:47:15.103] kestrel: kestrel-thrift server started on /0.0.0.0:2229
INF [20140406-01:47:15.108] kestrel: Starting up background expiration task.
INF [20140406-01:47:15.109] kestrel: Starting background-expiration
INF [20140406-01:47:15.113] kestrel: Kestrel 2.4.1 started.
INF [20140406-01:59:11.905] kestrel: Setting up queue PEP: maxItems=2147483647 maxSize=9223372036854775807.bytes maxItemSize=9223372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=134217728.bytes maxJournalSize=1073741824.bytes discardOldWhenFull=false keepJournal=true syncJournal=100.milliseconds expireToQueue=None maxExpireSweep=2147483647 fanoutOnly=false maxQueueAge=None (via ip-172-31-40-91.us-west-2.compute.internal:32901)
INF [20140406-01:59:11.968] kestrel: Replaying transaction journal for 'PEP'
INF [20140406-01:59:12.000] kestrel: No transaction journal for 'PEP'; starting with empty queue.
INF [20140406-01:59:12.001] kestrel: Finished transaction journal for 'PEP' (0 items, 0 bytes) xid=0
INF [20140406-01:59:12.095] kestrel: Setting up queue PR: maxItems=2147483647 maxSize=9223372036854775807.bytes maxItemSize=9223372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=134217728.bytes maxJournalSize=1073741824.bytes discardOldWhenFull=false keepJournal=true syncJournal=100.milliseconds expireToQueue=None maxExpireSweep=2147483647 fanoutOnly=false maxQueueAge=None (via ip-172-31-40-91.us-west-2.compute.internal:32902)
INF [20140406-01:59:12.099] kestrel: Replaying transaction journal for 'PR'
INF [20140406-01:59:12.101] kestrel: No transaction journal for 'PR'; starting with empty queue.
INF [20140406-01:59:12.101] kestrel: Finished transaction journal for 'PR' (0 items, 0 bytes) xid=0
INF [20140406-01:59:12.102] kestrel: Setting up queue SERVER: maxItems=2147483647 maxSize=9223372036854775807.bytes maxItemSize=9223372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=134217728.bytes maxJournalSize=1073741824.bytes discardOldWhenFull=false keepJournal=true syncJournal=100.milliseconds expireToQueue=None maxExpireSweep=2147483647 fanoutOnly=false maxQueueAge=None (via ip-172-31-40-91.us-west-2.compute.internal:32903)
INF [20140406-01:59:12.104] kestrel: Replaying transaction journal for 'SERVER'
INF [20140406-01:59:12.105] kestrel: No transaction journal for 'SERVER'; starting with empty queue.
INF [20140406-01:59:12.106] kestrel: Finished transaction journal for 'SERVER' (0 items, 0 bytes) xid=0
INF [20140406-01:59:12.147] kestrel: Setting up queue SERVICE: maxItems=2147483647 maxSize=9223372036854775807.bytes maxItemSize=9223372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=134217728.bytes maxJournalSize=1073741824.bytes discardOldWhenFull=false keepJournal=true syncJournal=100.milliseconds expireToQueue=None maxExpireSweep=2147483647 fanoutOnly=false maxQueueAge=None (via ip-172-31-40-91.us-west-2.compute.internal:32906)
INF [20140406-01:59:12.148] kestrel: Replaying transaction journal for 'SERVICE'
INF [20140406-01:59:12.153] kestrel: No transaction journal for 'SERVICE'; starting with empty queue.
INF [20140406-01:59:12.154] kestrel: Finished transaction journal for 'SERVICE' (0 items, 0 bytes) xid=0
FAT [20140407-14:24:57.950] kestrel-memcache: A server service  threw an exception
FAT [20140407-14:24:57.950] kestrel-memcache: java.lang.IndexOutOfBoundsException: 1
FAT [20140407-14:24:57.950] kestrel-memcache:     at scala.collection.LinearSeqOptimized$class.apply(LinearSeqOptimized.scala:51)
FAT [20140407-14:24:57.950] kestrel-memcache:     at scala.collection.immutable.List.apply(List.scala:76)
FAT [20140407-14:24:57.950] kestrel-memcache:     at net.lag.kestrel.MemcacheHandler.handle(MemcacheHandler.scala:117)
FAT [20140407-14:24:57.950] kestrel-memcache:     at net.lag.kestrel.MemcacheHandler.apply(MemcacheHandler.scala:63)
FAT [20140407-14:24:57.950] kestrel-memcache:     at net.lag.kestrel.MemcacheHandler.apply(MemcacheHandler.scala:34)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$1.apply(Filter.scala:111)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.service.StatsFilter.apply(StatsFilter.scala:24)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$2.apply(Filter.scala:52)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$4.apply(Filter.scala:69)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Service$$anon$1.apply(Service.scala:14)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.filter.MkJvmFilter$$anon$1.apply(JvmFilter.scala:19)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.tracing.TracingFilter$$anonfun$apply$1.apply(TracingFilter.scala:21)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.tracing.TracingFilter$$anonfun$apply$1.apply(TracingFilter.scala:16)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.tracing.Trace$.unwind(Trace.scala:128)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.tracing.TracingFilter.apply(TracingFilter.scala:16)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.Filter$$anon$2$$anon$3.apply(Filter.scala:53)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.filter.MonitorFilter$$anonfun$apply$1.apply(MonitorFilter.scala:16)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.finagle.filter.MonitorFilter$$anonfun$apply$1.apply(MonitorFilter.scala:16)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Future$$anonfun$monitored$1.apply$mcV$sp(Future.scala:128)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anonfun$apply$1.apply$mcV$sp(Monitor.scala:39)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:39)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anonfun$apply$1.apply(Monitor.scala:39)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anonfun$using$1.apply(Monitor.scala:104)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$.restoring(Monitor.scala:111)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$.using(Monitor.scala:102)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$class.apply(Monitor.scala:38)
FAT [20140407-14:24:57.950] kestrel-memcache:     at com.twitter.util.Monitor$$anon$1.apply(Monitor.scala:132)
FAT [20140407-14:24:57.950] kestrel-memcache:     (...more...)