mtomko
commented
8 years ago Sadly, my terminal history did not preserve the entire thread dump, but I think most of the relevant details are still here. I'll try to capture a full one next time but for now:
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #19" #274 daemon prio=5 os_prio=31 tid=0x00007ff917e1a000 nid=0xc203 runnable [0x0000700004690000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000753096708> (a sun.nio.ch.Util$2)
- locked <0x00000007530966f8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007530965d8> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #18" #273 daemon prio=5 os_prio=31 tid=0x00007ff90c92d800 nid=0x6427 runnable [0x0000700004284000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000753094780> (a sun.nio.ch.Util$2)
- locked <0x0000000753094770> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000753094540> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Hashed wheel timer #1" #272 prio=5 os_prio=31 tid=0x00007ff916dcf000 nid=0xa513 waiting on condition [0x000070000407e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)
"oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser" #265 daemon prio=4 os_prio=31 tid=0x00007ff90c5c5800 nid=0x6827 in Object.wait() [0x000070000242a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:327)
- locked <0x0000000752f0b038> (a oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser)
"ForkJoinPool-3-worker-11" #250 daemon prio=5 os_prio=31 tid=0x00007ff90f0cf800 nid=0x5a27 waiting on condition [0x0000700003f7b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074e05b3d0> (a scala.concurrent.forkjoin.ForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"Scala Presentation Compiler []" #226 daemon prio=5 os_prio=31 tid=0x00007ff90f283800 nid=0x7923 in Object.wait() [0x00007000018de000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at scala.tools.nsc.util.WorkScheduler.waitForMoreWork(WorkScheduler.scala:16)
- locked <0x000000074b695fa0> (a scala.tools.nsc.util.WorkScheduler)
at scala.tools.nsc.interactive.PresentationCompilerThread$$anonfun$run$1.apply$mcZ$sp(PresentationCompilerThread.scala:21)
at scala.tools.nsc.io.NullLogger$.logreplay(Replayer.scala:38)
at scala.tools.nsc.interactive.PresentationCompilerThread.run(PresentationCompilerThread.scala:21)
"New I/O server boss #17" #215 prio=5 os_prio=31 tid=0x00007ff913bea000 nid=0xad17 runnable [0x0000700003763000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074aaf78a8> (a sun.nio.ch.Util$2)
- locked <0x000000074aaf7898> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074aaf7418> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #16" #214 prio=5 os_prio=31 tid=0x00007ff91202c800 nid=0xb613 runnable [0x0000700003660000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a759698> (a sun.nio.ch.Util$2)
- locked <0x000000074a759688> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a759568> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #15" #213 prio=5 os_prio=31 tid=0x00007ff90c8d0000 nid=0x7e0b runnable [0x000070000355d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a74d420> (a sun.nio.ch.Util$2)
- locked <0x000000074a74d410> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a74d2f0> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #14" #212 prio=5 os_prio=31 tid=0x00007ff912f55000 nid=0xa207 runnable [0x000070000345a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a755840> (a sun.nio.ch.Util$2)
- locked <0x000000074a755830> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a755710> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #13" #211 prio=5 os_prio=31 tid=0x00007ff912f54800 nid=0xa107 runnable [0x0000700003357000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a755230> (a sun.nio.ch.Util$2)
- locked <0x000000074a755220> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a755100> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #12" #210 prio=5 os_prio=31 tid=0x00007ff90ce74000 nid=0xa007 runnable [0x0000700003254000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a759068> (a sun.nio.ch.Util$2)
- locked <0x000000074a759058> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a758f38> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #11" #209 prio=5 os_prio=31 tid=0x00007ff90cdca000 nid=0x9007 runnable [0x0000700003151000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a758448> (a sun.nio.ch.Util$2)
- locked <0x000000074a758438> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a758318> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #10" #208 prio=5 os_prio=31 tid=0x00007ff913cef000 nid=0x8607 runnable [0x000070000304e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a754610> (a sun.nio.ch.Util$2)
- locked <0x000000074a754600> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a7544e0> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #9" #207 prio=5 os_prio=31 tid=0x00007ff913cff800 nid=0x8e07 runnable [0x0000700002f4b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a758a58> (a sun.nio.ch.Util$2)
- locked <0x000000074a758a48> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a758928> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #8" #206 prio=5 os_prio=31 tid=0x00007ff914135000 nid=0x9507 runnable [0x0000700002e48000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a754c20> (a sun.nio.ch.Util$2)
- locked <0x000000074a754c10> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a754af0> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #7" #205 prio=5 os_prio=31 tid=0x00007ff912df5800 nid=0x9407 runnable [0x0000700002d45000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a74da30> (a sun.nio.ch.Util$2)
- locked <0x000000074a74da20> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a74d900> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #6" #204 prio=5 os_prio=31 tid=0x00007ff912f56800 nid=0x8207 runnable [0x0000700002c42000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a74ce10> (a sun.nio.ch.Util$2)
- locked <0x000000074a74ce00> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a74cce0> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #5" #203 prio=5 os_prio=31 tid=0x00007ff90cfed800 nid=0x9e07 runnable [0x0000700002b3f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a755e50> (a sun.nio.ch.Util$2)
- locked <0x000000074a755e40> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a755d20> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #4" #202 prio=5 os_prio=31 tid=0x00007ff913cfd800 nid=0x9c07 runnable [0x0000700002a3c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a759ca8> (a sun.nio.ch.Util$2)
- locked <0x000000074a759c98> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a759b78> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #3" #201 prio=5 os_prio=31 tid=0x00007ff913dbf000 nid=0x8b07 runnable [0x0000700002939000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a74c800> (a sun.nio.ch.Util$2)
- locked <0x000000074a74c7f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a74c6d0> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #2" #200 prio=5 os_prio=31 tid=0x00007ff90cc67000 nid=0x8a07 runnable [0x0000700002836000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074a74e040> (a sun.nio.ch.Util$2)
- locked <0x000000074a74e030> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a74df10> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"New I/O worker #1" #199 prio=5 os_prio=31 tid=0x00007ff90cfde800 nid=0x8807 runnable [0x0000700002733000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000074aaf7a18> (a sun.nio.ch.Util$2)
- locked <0x000000074aaf7a08> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000074a7f1318> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"application-akka.actor.default-dispatcher-3" #197 prio=5 os_prio=31 tid=0x00007ff90e046000 nid=0x990b waiting on condition [0x000070000252d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074aa6fe98> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"application-scheduler-1" #195 prio=5 os_prio=31 tid=0x00007ff90f354800 nid=0x8513 sleeping[0x0000700002327000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)
"pool-19-thread-5" #190 prio=5 os_prio=31 tid=0x00007ff912101000 nid=0x6637 runnable [0x0000700001ce9000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.read0(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:207)
at jline.internal.NonBlockingInputStream.read(NonBlockingInputStream.java:245)
at jline.internal.InputStreamReader.read(InputStreamReader.java:257)
- locked <0x000000074b6bda20> (a jline.internal.NonBlockingInputStream)
at jline.internal.InputStreamReader.read(InputStreamReader.java:194)
- locked <0x000000074b6bda20> (a jline.internal.NonBlockingInputStream)
at jline.console.ConsoleReader.readCharacter(ConsoleReader.java:2136)
at jline.console.ConsoleReader.readCharacter(ConsoleReader.java:2126)
at play.sbt.PlayConsoleInteractionMode$$anonfun$waitForKey$1.play$sbt$PlayConsoleInteractionMode$$anonfun$$waitEOF$1(PlayInteractionMode.scala:62)
at play.sbt.PlayConsoleInteractionMode$$anonfun$waitForKey$1$$anonfun$apply$1.apply$mcV$sp(PlayInteractionMode.scala:73)
at play.sbt.PlayConsoleInteractionMode$$anonfun$doWithoutEcho$1.apply(PlayInteractionMode.scala:80)
at play.sbt.PlayConsoleInteractionMode$$anonfun$doWithoutEcho$1.apply(PlayInteractionMode.scala:77)
at play.sbt.PlayConsoleInteractionMode$.withConsoleReader(PlayInteractionMode.scala:57)
at play.sbt.PlayConsoleInteractionMode$.doWithoutEcho(PlayInteractionMode.scala:77)
at play.sbt.PlayConsoleInteractionMode$$anonfun$waitForKey$1.apply(PlayInteractionMode.scala:73)
at play.sbt.PlayConsoleInteractionMode$$anonfun$waitForKey$1.apply(PlayInteractionMode.scala:60)
at play.sbt.PlayConsoleInteractionMode$.withConsoleReader(PlayInteractionMode.scala:57)
at play.sbt.PlayConsoleInteractionMode$.waitForKey(PlayInteractionMode.scala:60)
at play.sbt.PlayConsoleInteractionMode$.waitForCancel(PlayInteractionMode.scala:83)
at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.apply(PlayRun.scala:123)
at play.sbt.run.PlayRun$$anonfun$playRunTask$1$$anonfun$apply$2$$anonfun$apply$3.apply(PlayRun.scala:53)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"sbt-web-akka.actor.default-dispatcher-2" #177 prio=5 os_prio=31 tid=0x00007ff913da5800 nid=0x5b2f waiting on condition [0x00007000016d8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007496e8080> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"sbt-web-scheduler-1" #176 prio=5 os_prio=31 tid=0x00007ff90f5cf000 nid=0x6b23 sleeping[0x00007000015d5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(Scheduler.scala:226)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:405)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:745)
"oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser" #120 daemon prio=4 os_prio=31 tid=0x00007ff90ce06000 nid=0xb30b in Object.wait() [0x0000700003e78000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:327)
- locked <0x0000000744408480> (a oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser)
"application-akka.actor.default-dispatcher-5" #118 prio=5 os_prio=31 tid=0x00007ff90f448000 nid=0xb00b waiting on condition [0x0000700003c72000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074455ebb0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"application-akka.actor.default-dispatcher-4" #117 prio=5 os_prio=31 tid=0x00007ff90fd20000 nid=0xb10b waiting on condition [0x0000700003b6f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074455ebb0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"application-akka.actor.default-dispatcher-3" #116 prio=5 os_prio=31 tid=0x00007ff90fd1b800 nid=0xb413 waiting on condition [0x0000700003a6c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074455ebb0> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"application-akka.actor.default-dispatcher-2" #115 prio=5 os_prio=31 tid=0x00007ff90f5a6800 nid=0xa60b waiting on condition [0x0000700003966000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.broadinstitute.rnai.scalamari.retry.package$ExponentialBackoffRetryStrategy.shouldRetry(retry-package.scala:115)
at org.broadinstitute.rnai.scalamari.retry.package$.retry$1(retry-package.scala:161)
at org.broadinstitute.rnai.scalamari.retry.package$.withRetry(retry-package.scala:169)
at org.broadinstitute.rnai.scalamari.file.package$.withLock(file-package.scala:252)
at org.broadinstitute.gpp.inferno.agents.crispr.proximitysearch.LocusProximitySearchMonitor.maybeCacheLoci(LocusProximitySearchMonitor.scala:47)
at org.broadinstitute.gpp.inferno.agents.crispr.proximitysearch.LocusProximitySearchMonitor.computeValue(LocusProximitySearchMonitor.scala:31)
at org.broadinstitute.gpp.inferno.agents.crispr.proximitysearch.LocusProximitySearchMonitor.computeValue(LocusProximitySearchMonitor.scala:20)
at org.broadinstitute.gpp.inferno.agents.ResourceMonitor.<init>(Agent.scala:169)
at org.broadinstitute.gpp.inferno.agents.DataSourceMonitor.<init>(Agent.scala:233)
at org.broadinstitute.gpp.inferno.agents.refdb.RefDbVintageMonitor.<init>(RefdbVintageMonitor.scala:15)
at org.broadinstitute.gpp.inferno.agents.crispr.proximitysearch.LocusProximitySearchMonitor.<init>(LocusProximitySearchMonitor.scala:25)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents$$anonfun$9.apply(RefDbAgents.scala:82)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents$$anonfun$9.apply(RefDbAgents.scala:82)
at scaldi.LazyBinding$$anonfun$target$2.apply(Binding.scala:180)
at scaldi.LazyBinding$$anonfun$target$2.apply(Binding.scala:180)
at scala.Option.map(Option.scala:146)
at scaldi.LazyBinding.target$lzycompute(Binding.scala:180)
- locked <0x000000074469d5f8> (a scaldi.LazyBinding)
at scaldi.LazyBinding.target(Binding.scala:180)
at scaldi.LazyBinding$$anonfun$get$2.apply(Binding.scala:189)
at scaldi.LazyBinding$$anonfun$get$2.apply(Binding.scala:188)
at scala.Option.foreach(Option.scala:257)
at scaldi.LazyBinding.get(Binding.scala:188)
at scaldi.Binding$$anon$1.get(Binding.scala:66)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scala.Option.flatMap(Option.scala:171)
at scaldi.Injectable$class.injectWithDefault(Injectable.scala:152)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents.injectWithDefault(RefDbAgents.scala:46)
at scaldi.Injectable$$anonfun$inject$2.apply(Injectable.scala:71)
at scaldi.Injectable$$anonfun$inject$2.apply(Injectable.scala:71)
at scaldi.util.Util$WorkflowHelper$.$bar$greater$extension(Util.scala:7)
at scaldi.Injectable$class.inject(Injectable.scala:71)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents.inject(RefDbAgents.scala:46)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents$$anonfun$24.apply(RefDbAgents.scala:180)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents$$anonfun$24.apply(RefDbAgents.scala:177)
at scaldi.LazyBinding$$anonfun$target$2.apply(Binding.scala:180)
at scaldi.LazyBinding$$anonfun$target$2.apply(Binding.scala:180)
at scala.Option.map(Option.scala:146)
at scaldi.LazyBinding.target$lzycompute(Binding.scala:180)
- locked <0x000000074469d7e8> (a scaldi.LazyBinding)
at scaldi.LazyBinding.target(Binding.scala:180)
at scaldi.LazyBinding.get(Binding.scala:196)
at scaldi.Binding$$anon$1.get(Binding.scala:66)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scala.Option.flatMap(Option.scala:171)
at scaldi.Injectable$class.injectWithDefault(Injectable.scala:152)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents.injectWithDefault(RefDbAgents.scala:46)
at scaldi.Injectable$$anonfun$inject$2.apply(Injectable.scala:71)
at scaldi.Injectable$$anonfun$inject$2.apply(Injectable.scala:71)
at scaldi.util.Util$WorkflowHelper$.$bar$greater$extension(Util.scala:7)
at scaldi.Injectable$class.inject(Injectable.scala:71)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents.inject(RefDbAgents.scala:46)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents$$anonfun$33.apply(RefDbAgents.scala:228)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents$$anonfun$33.apply(RefDbAgents.scala:224)
at scaldi.LazyBinding$$anonfun$target$2.apply(Binding.scala:180)
at scaldi.LazyBinding$$anonfun$target$2.apply(Binding.scala:180)
at scala.Option.map(Option.scala:146)
at scaldi.LazyBinding.target$lzycompute(Binding.scala:180)
- locked <0x000000074469d948> (a scaldi.LazyBinding)
at scaldi.LazyBinding.target(Binding.scala:180)
at scaldi.LazyBinding.get(Binding.scala:196)
at scaldi.Binding$$anon$1.get(Binding.scala:66)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scala.Option.flatMap(Option.scala:171)
at scaldi.Injectable$class.injectWithDefault(Injectable.scala:152)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents.injectWithDefault(RefDbAgents.scala:46)
at scaldi.Injectable$$anonfun$inject$2.apply(Injectable.scala:71)
at scaldi.Injectable$$anonfun$inject$2.apply(Injectable.scala:71)
at scaldi.util.Util$WorkflowHelper$.$bar$greater$extension(Util.scala:7)
at scaldi.Injectable$class.inject(Injectable.scala:71)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents.inject(RefDbAgents.scala:46)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents$$anonfun$36.apply(RefDbAgents.scala:256)
at org.broadinstitute.gpp.di.inferno.agents.RefDbAgents$$anonfun$36.apply(RefDbAgents.scala:252)
at scaldi.LazyBinding$$anonfun$target$2.apply(Binding.scala:180)
at scaldi.LazyBinding$$anonfun$target$2.apply(Binding.scala:180)
at scala.Option.map(Option.scala:146)
at scaldi.LazyBinding.target$lzycompute(Binding.scala:180)
- locked <0x000000074469da80> (a scaldi.LazyBinding)
at scaldi.LazyBinding.target(Binding.scala:180)
at scaldi.LazyBinding.get(Binding.scala:196)
at scaldi.Binding$$anon$1.get(Binding.scala:66)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scala.Option.flatMap(Option.scala:171)
at scaldi.Injectable$class.injectWithDefault(Injectable.scala:152)
at controllers.construct.Constructs.injectWithDefault(Constructs.scala:33)
at scaldi.Injectable$$anonfun$inject$2.apply(Injectable.scala:71)
at scaldi.Injectable$$anonfun$inject$2.apply(Injectable.scala:71)
at scaldi.util.Util$WorkflowHelper$.$bar$greater$extension(Util.scala:7)
at scaldi.Injectable$class.inject(Injectable.scala:71)
at controllers.construct.Constructs.inject(Constructs.scala:33)
at controllers.construct.Constructs.humanThreatMatrician$lzycompute(Constructs.scala:40)
- locked <0x000000074469dae8> (a controllers.construct.Constructs)
at controllers.construct.Constructs.humanThreatMatrician(Constructs.scala:40)
at controllers.construct.Constructs$$anonfun$controllers$construct$Constructs$$sgrnaDetailPage$1.apply(Constructs.scala:134)
at controllers.construct.Constructs$$anonfun$controllers$construct$Constructs$$sgrnaDetailPage$1.apply(Constructs.scala:126)
at org.broadinstitute.possum.package$.withSession(package.scala:56)
at org.broadinstitute.possum.package$.withSession(package.scala:43)
at controllers.construct.Constructs.controllers$construct$Constructs$$sgrnaDetailPage(Constructs.scala:126)
at controllers.construct.Constructs$$anonfun$details$1$$anonfun$apply$1$$anonfun$apply$4.apply(Constructs.scala:58)
at controllers.construct.Constructs$$anonfun$details$1$$anonfun$apply$1$$anonfun$apply$4.apply(Constructs.scala:52)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"FSEvent thread" #64 daemon prio=5 os_prio=31 tid=0x00007ff913a79000 nid=0x6717 runnable [0x0000700002121000]
java.lang.Thread.State: RUNNABLE
at net.contentobjects.jnotify.macosx.JNotify_macosx.nativeNotifyLoop(Native Method)
at net.contentobjects.jnotify.macosx.JNotify_macosx.access$400(Unknown Source)
at net.contentobjects.jnotify.macosx.JNotify_macosx$1.run(Unknown Source)
"Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007ff90c834000 nid=0x5203 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007ff90c038000 nid=0x5003 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007ff90c027800 nid=0x4e03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007ff90c026800 nid=0x4c03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007ff90e818000 nid=0x4a03 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007ff90e817800 nid=0x3e0f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007ff90c801000 nid=0x3803 in Object.wait() [0x0000700000d3a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x000000074000aa98> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007ff90c013800 nid=0x3603 in Object.wait() [0x0000700000c37000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000740012e10> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=31 tid=0x00007ff90e001800 nid=0xc07 waiting on condition [0x0000700000218000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007490d9410> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)
at sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:188)
at sbt.Execute.next$1(Execute.scala:85)
at sbt.Execute.processAll(Execute.scala:88)
at sbt.Execute.runKeep(Execute.scala:68)
at sbt.EvaluateTask$.liftedTree1$1(EvaluateTask.scala:359)
at sbt.EvaluateTask$.run$1(EvaluateTask.scala:358)
at sbt.EvaluateTask$.runTask(EvaluateTask.scala:378)
at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:64)
at sbt.Aggregation$$anonfun$3.apply(Aggregation.scala:62)
at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:314)
at sbt.Aggregation$.timedRun(Aggregation.scala:62)
at sbt.Aggregation$.runTasks(Aggregation.scala:71)
at sbt.Aggregation$$anonfun$applyDynamicTasks$1.apply(Aggregation.scala:112)
at sbt.Aggregation$$anonfun$applyDynamicTasks$1.apply(Aggregation.scala:110)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:61)
at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:61)
at sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:244)
at sbt.Act$$anonfun$sbt$Act$$actParser0$1$$anonfun$sbt$Act$$anonfun$$evaluate$1$1$$anonfun$apply$10.apply(Act.scala:241)
at sbt.Command$.process(Command.scala:93)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:96)
at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:96)
at sbt.State$$anon$1.process(State.scala:184)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:96)
at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:96)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.MainLoop$.next(MainLoop.scala:96)
at sbt.MainLoop$.run(MainLoop.scala:89)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:68)
at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:63)
at sbt.Using.apply(Using.scala:24)
at sbt.MainLoop$.runWithNewLog(MainLoop.scala:63)
at sbt.MainLoop$.runAndClearLast(MainLoop.scala:46)
at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:30)
at sbt.MainLoop$.runLogged(MainLoop.scala:22)
at sbt.StandardMain$.runManaged(Main.scala:54)
at sbt.xMain.run(Main.scala:29)
at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
at xsbt.boot.Launch$.run(Launch.scala:109)
at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
at xsbt.boot.Launch$.launch(Launch.scala:117)
at xsbt.boot.Launch$.apply(Launch.scala:18)
at xsbt.boot.Boot$.runImpl(Boot.scala:41)
at xsbt.boot.Boot$.main(Boot.scala:17)
at xsbt.boot.Boot.main(Boot.scala)
"VM Thread" os_prio=31 tid=0x00007ff90e017800 nid=0x3403 runnable
"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007ff90c805800 nid=0x1b07 runnable
"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007ff90c012000 nid=0x260b runnable
"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007ff90c012800 nid=0x2803 runnable
"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007ff90c013000 nid=0x2a03 runnable
"GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007ff90e00a800 nid=0x2c03 runnable
"GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007ff90c806000 nid=0x2e03 runnable
"GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007ff90c806800 nid=0x3003 runnable
"GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007ff90c807000 nid=0x3203 runnable
"VM Periodic Task Thread" os_prio=31 tid=0x00007ff90c05b800 nid=0x5403 waiting on condition
JNI global references: 666
Found one Java-level deadlock:
=============================
"application-akka.actor.default-dispatcher-8":
waiting to lock monitor 0x00007ff917a0f958 (object 0x00000007a0ca8b00, a scaldi.jsr330.AnnotationBinding),
which is held by "application-akka.actor.default-dispatcher-7"
"application-akka.actor.default-dispatcher-7":
waiting to lock monitor 0x00007ff90ec03728 (object 0x000000079cc2f328, a scaldi.jsr330.OnDemandAnnotationInjector),
which is held by "application-akka.actor.default-dispatcher-8"
Java stack information for the threads listed above:
===================================================
"application-akka.actor.default-dispatcher-8":
at scaldi.jsr330.AnnotationBinding.resolvedCondition$lzycompute(AnnotationBinding.scala:15)
- waiting to lock <0x00000007a0ca8b00> (a scaldi.jsr330.AnnotationBinding)
at scaldi.jsr330.AnnotationBinding.resolvedCondition(AnnotationBinding.scala:15)
at scaldi.Identifiable$class.isDefinedFor(Binding.scala:23)
at scaldi.jsr330.AnnotationBinding.isDefinedFor(AnnotationBinding.scala:15)
at scaldi.jsr330.OnDemandAnnotationInjector$$anonfun$getBindingInternal$2$$anonfun$apply$1.apply(OnDemandAnnotationInjector.scala:19)
at scaldi.jsr330.OnDemandAnnotationInjector$$anonfun$getBindingInternal$2$$anonfun$apply$1.apply(OnDemandAnnotationInjector.scala:19)
at scala.collection.LinearSeqOptimized$class.find(LinearSeqOptimized.scala:113)
at scala.collection.immutable.List.find(List.scala:84)
at scaldi.jsr330.OnDemandAnnotationInjector$$anonfun$getBindingInternal$2.apply(OnDemandAnnotationInjector.scala:19)
- locked <0x000000079cc2f328> (a scaldi.jsr330.OnDemandAnnotationInjector)
at scaldi.jsr330.OnDemandAnnotationInjector$$anonfun$getBindingInternal$2.apply(OnDemandAnnotationInjector.scala:18)
at scala.Option.orElse(Option.scala:289)
at scaldi.jsr330.OnDemandAnnotationInjector.getBindingInternal(OnDemandAnnotationInjector.scala:17)
at scaldi.MutableInjectorAggregation$$anonfun$getBindingInternal$2.apply(Injector.scala:85)
at scaldi.MutableInjectorAggregation$$anonfun$getBindingInternal$2.apply(Injector.scala:84)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.TraversableOnce$class.collectFirst(TraversableOnce.scala:145)
at scala.collection.SeqViewLike$AbstractTransformed.collectFirst(SeqViewLike.scala:37)
at scaldi.MutableInjectorAggregation.getBindingInternal(Injector.scala:87)
at scaldi.InjectorWithLifecycle$$anonfun$getBinding$3.apply(Injector.scala:170)
at scaldi.InjectorWithLifecycle$$anonfun$getBinding$3.apply(Injector.scala:170)
at scaldi.util.Util$WorkflowHelper$.$bar$greater$extension(Util.scala:7)
at scaldi.InjectorWithLifecycle$class.getBinding(Injector.scala:170)
at scaldi.MutableInjectorAggregation.getBinding(Injector.scala:78)
at scaldi.Injectable$class.injectWithDefault(Injectable.scala:152)
at scaldi.play.ScaldiBuilder$.injectWithDefault(ScaldiBuilder.scala:128)
at scaldi.play.ScaldiBuilder$$anonfun$6$$anonfun$apply$1$$anonfun$apply$12.apply(ScaldiBuilder.scala:227)
at scaldi.ProviderBinding.target(Binding.scala:217)
at scaldi.ProviderBinding.get(Binding.scala:223)
at scaldi.ProviderBinding.get(Binding.scala:213)
at scaldi.Binding$$anon$1.get(Binding.scala:66)
at scaldi.play.ScaldiInjector$$anonfun$scaldi$play$ScaldiInjector$$getActualBinding$1$$anonfun$apply$2.apply(ScaldiInjector.scala:42)
at scaldi.play.ScaldiInjector$$anonfun$scaldi$play$ScaldiInjector$$getActualBinding$1$$anonfun$apply$2.apply(ScaldiInjector.scala:42)
at scala.Option.map(Option.scala:146)
at scaldi.play.ScaldiInjector$$anonfun$scaldi$play$ScaldiInjector$$getActualBinding$1.apply(ScaldiInjector.scala:42)
at scaldi.play.ScaldiInjector$$anonfun$scaldi$play$ScaldiInjector$$getActualBinding$1.apply(ScaldiInjector.scala:38)
at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
at scaldi.play.ScaldiInjector.scaldi$play$ScaldiInjector$$getActualBinding(ScaldiInjector.scala:38)
at scaldi.play.ScaldiInjector$$anonfun$instanceOf$1.apply(ScaldiInjector.scala:23)
at scaldi.play.ScaldiInjector$$anonfun$instanceOf$1.apply(ScaldiInjector.scala:22)
at scala.Option.getOrElse(Option.scala:121)
at scaldi.play.ScaldiInjector.instanceOf(ScaldiInjector.scala:22)
at scaldi.play.ScaldiInjector.instanceOf(ScaldiInjector.scala:18)
at scaldi.play.ScaldiInjector.instanceOf(ScaldiInjector.scala:15)
at play.api.Application$$anonfun$instanceCache$1.apply(Application.scala:234)
at play.api.Application$$anonfun$instanceCache$1.apply(Application.scala:234)
at play.utils.InlineCache.fresh(InlineCache.scala:69)
at play.utils.InlineCache.apply(InlineCache.scala:60)
at play.api.db.DB$.db(DB.scala:22)
at play.api.db.DB$.withConnection(DB.scala:61)
at models.HistoryEntry$$anonfun$findMostRecent$1.apply(HistoryEntry.scala:161)
at models.HistoryEntry$$anonfun$findMostRecent$1.apply(HistoryEntry.scala:161)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:169)
at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)
at scala.concurrent.package$.blocking(package.scala:123)
at models.HistoryEntry$.findMostRecent(HistoryEntry.scala:160)
at models.HistoryEntry.insert(HistoryEntry.scala:37)
at controllers.tools.DataTableMegaWidgetController$WithHistoryLogging$$anonfun$apply$1.apply$mcV$sp(DataTableMegaWidgetController.scala:130)
at controllers.tools.DataTableMegaWidgetController$WithHistoryLogging$$anonfun$apply$1.apply(DataTableMegaWidgetController.scala:112)
at controllers.tools.DataTableMegaWidgetController$WithHistoryLogging$$anonfun$apply$1.apply(DataTableMegaWidgetController.scala:112)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"application-akka.actor.default-dispatcher-7":
at scaldi.jsr330.OnDemandAnnotationInjector$$anonfun$getBindingInternal$2.apply(OnDemandAnnotationInjector.scala:19)
- waiting to lock <0x000000079cc2f328> (a scaldi.jsr330.OnDemandAnnotationInjector)
at scaldi.jsr330.OnDemandAnnotationInjector$$anonfun$getBindingInternal$2.apply(OnDemandAnnotationInjector.scala:18)
at scala.Option.orElse(Option.scala:289)
at scaldi.jsr330.OnDemandAnnotationInjector.getBindingInternal(OnDemandAnnotationInjector.scala:17)
at scaldi.MutableInjectorAggregation$$anonfun$getBindingInternal$2.apply(Injector.scala:85)
at scaldi.MutableInjectorAggregation$$anonfun$getBindingInternal$2.apply(Injector.scala:84)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.TraversableOnce$class.collectFirst(TraversableOnce.scala:145)
at scala.collection.SeqViewLike$AbstractTransformed.collectFirst(SeqViewLike.scala:37)
at scaldi.MutableInjectorAggregation.getBindingInternal(Injector.scala:87)
at scaldi.InjectorWithLifecycle$$anonfun$getBinding$3.apply(Injector.scala:170)
at scaldi.InjectorWithLifecycle$$anonfun$getBinding$3.apply(Injector.scala:170)
at scaldi.util.Util$WorkflowHelper$.$bar$greater$extension(Util.scala:7)
at scaldi.InjectorWithLifecycle$class.getBinding(Injector.scala:170)
at scaldi.MutableInjectorAggregation.getBinding(Injector.scala:78)
at scaldi.Injectable$class.injectWithDefault(Injectable.scala:152)
at scaldi.play.ScaldiBuilder$.injectWithDefault(ScaldiBuilder.scala:128)
at scaldi.play.ScaldiBuilder$$anonfun$6$$anonfun$apply$1$$anonfun$apply$18.apply(ScaldiBuilder.scala:240)
at scaldi.ProviderBinding.target(Binding.scala:217)
at scaldi.ProviderBinding.get(Binding.scala:223)
at scaldi.ProviderBinding.get(Binding.scala:213)
at scaldi.Binding$$anon$1.get(Binding.scala:66)
at scaldi.jsr330.AnnotationBinding$$anonfun$scaldi$jsr330$AnnotationBinding$$injectSymbol$2.apply(AnnotationBinding.scala:171)
at scaldi.jsr330.AnnotationBinding$$anonfun$scaldi$jsr330$AnnotationBinding$$injectSymbol$2.apply(AnnotationBinding.scala:171)
at scala.Option.flatMap(Option.scala:171)
at scaldi.jsr330.AnnotationBinding.scaldi$jsr330$AnnotationBinding$$injectSymbol(AnnotationBinding.scala:171)
at scaldi.jsr330.AnnotationBinding$$anonfun$22.apply(AnnotationBinding.scala:132)
at scaldi.jsr330.AnnotationBinding$$anonfun$22.apply(AnnotationBinding.scala:132)
at scala.collection.immutable.List.map(List.scala:277)
at scaldi.jsr330.AnnotationBinding.scaldi$jsr330$AnnotationBinding$$createNewInstance(AnnotationBinding.scala:132)
at scaldi.jsr330.AnnotationBinding$$anonfun$2$$anonfun$apply$2.apply(AnnotationBinding.scala:41)
at scaldi.jsr330.AnnotationBinding$$anonfun$2$$anonfun$apply$2.apply(AnnotationBinding.scala:41)
at scaldi.jsr330.AnnotationBinding.initNewInstance(AnnotationBinding.scala:119)
at scaldi.jsr330.AnnotationBinding.getInstance(AnnotationBinding.scala:104)
- locked <0x00000007a0ca8b00> (a scaldi.jsr330.AnnotationBinding)
at scaldi.jsr330.AnnotationBinding.get(AnnotationBinding.scala:85)
at scaldi.Binding$$anon$1.get(Binding.scala:66)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scaldi.Injectable$$anonfun$injectWithDefault$1.apply(Injectable.scala:152)
at scala.Option.flatMap(Option.scala:171)
at scaldi.Injectable$class.injectWithDefault(Injectable.scala:152)
at scaldi.play.ScaldiBuilder$.injectWithDefault(ScaldiBuilder.scala:128)
at scaldi.play.ScaldiBuilder$$anonfun$6$$anonfun$apply$1$$anonfun$apply$12.apply(ScaldiBuilder.scala:227)
at scaldi.ProviderBinding.target(Binding.scala:217)
at scaldi.ProviderBinding.get(Binding.scala:223)
at scaldi.ProviderBinding.get(Binding.scala:213)
at scaldi.Binding$$anon$1.get(Binding.scala:66)
at scaldi.play.ScaldiInjector$$anonfun$scaldi$play$ScaldiInjector$$getActualBinding$1$$anonfun$apply$2.apply(ScaldiInjector.scala:42)
at scaldi.play.ScaldiInjector$$anonfun$scaldi$play$ScaldiInjector$$getActualBinding$1$$anonfun$apply$2.apply(ScaldiInjector.scala:42)
at scala.Option.map(Option.scala:146)
at scaldi.play.ScaldiInjector$$anonfun$scaldi$play$ScaldiInjector$$getActualBinding$1.apply(ScaldiInjector.scala:42)
at scaldi.play.ScaldiInjector$$anonfun$scaldi$play$ScaldiInjector$$getActualBinding$1.apply(ScaldiInjector.scala:38)
at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
at scaldi.play.ScaldiInjector.scaldi$play$ScaldiInjector$$getActualBinding(ScaldiInjector.scala:38)
at scaldi.play.ScaldiInjector$$anonfun$instanceOf$1.apply(ScaldiInjector.scala:23)
at scaldi.play.ScaldiInjector$$anonfun$instanceOf$1.apply(ScaldiInjector.scala:22)
at scala.Option.getOrElse(Option.scala:121)
at scaldi.play.ScaldiInjector.instanceOf(ScaldiInjector.scala:22)
at scaldi.play.ScaldiInjector.instanceOf(ScaldiInjector.scala:18)
at scaldi.play.ScaldiInjector.instanceOf(ScaldiInjector.scala:15)
at play.api.Application$$anonfun$instanceCache$1.apply(Application.scala:234)
at play.api.Application$$anonfun$instanceCache$1.apply(Application.scala:234)
at play.utils.InlineCache.fresh(InlineCache.scala:69)
at play.utils.InlineCache.apply(InlineCache.scala:60)
at play.api.db.DB$.db(DB.scala:22)
at play.api.db.DB$.withConnection(DB.scala:61)
at models.HistoryEntry$$anonfun$findMostRecent$1.apply(HistoryEntry.scala:161)
at models.HistoryEntry$$anonfun$findMostRecent$1.apply(HistoryEntry.scala:161)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:169)
at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)
at scala.concurrent.package$.blocking(package.scala:123)
at models.HistoryEntry$.findMostRecent(HistoryEntry.scala:160)
at models.HistoryEntry.insert(HistoryEntry.scala:37)
at controllers.tools.DataTableMegaWidgetController$WithHistoryLogging$$anonfun$apply$1.apply$mcV$sp(DataTableMegaWidgetController.scala:130)
at controllers.tools.DataTableMegaWidgetController$WithHistoryLogging$$anonfun$apply$1.apply(DataTableMegaWidgetController.scala:112)
at controllers.tools.DataTableMegaWidgetController$WithHistoryLogging$$anonfun$apply$1.apply(DataTableMegaWidgetController.scala:112)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Found 1 deadlock.
Heap
PSYoungGen total 492544K, used 291581K [0x0000000795580000, 0x00000007c0000000, 0x00000007c0000000)
eden space 286208K, 72% used [0x0000000795580000,0x00000007a1f33fe0,0x00000007a6d00000)
from space 206336K, 41% used [0x00000007a6d00000,0x00000007ac00b7e0,0x00000007b3680000)
to space 199680K, 0% used [0x00000007b3d00000,0x00000007b3d00000,0x00000007c0000000)
ParOldGen total 1398272K, used 667198K [0x0000000740000000, 0x0000000795580000, 0x0000000795580000)
object space 1398272K, 47% used [0x0000000740000000,0x0000000768b8f9b0,0x0000000795580000)
Metaspace used 239142K, capacity 240793K, committed 241448K, reserved 1251328K
class space used 38372K, capacity 38848K, committed 38952K, reserved 1048576K
bilyush
dave-handy
zella
Hi there,
I've been trying to chase down an issue that appears to be a deadlock originating in the scaldi-jsr module. My application is a Play Framework site (play version 2.4.3) and we've recently converted it to use Scaldi for dependency injection. I have not yet demonstrated this problem in prod mode, but in dev mode I run into a deadlock every few refreshes. Looking at the thread dump, it looks like the code that's doing the hold and wait is in the scaldi-jsr330 module. According to my IDE, we're pulling in version 0.5.8 of the scaldi-jsr330 module. I'll summarize the evidence in this comment and post a full thread dump in a separate comment.
To begin with, the thread dump shows:
For the first thread:
Note that this thread has locked an instance of
OnDemandAnnotationInjectorand is waiting to lock aAnnotationBinding.Here's the other thread:
This thread holds an
AnnotationBindingand is waiting to lock theOnDemandAnnotationInjector.Deeper in the stack traces, both threads are executing userland code until they call
play.api.db.DB$.withConnection:Here,
models.HistoryEntry$$anonfun$findMostRecent$1.applyis code that we wrote; it's attempting to look up some information from our database.So something about the initialization state of the application at this point appears to lead to a race condition in the Scaldi injector resulting in a thread holding one resource and waiting to lock another while another thread attempts to do the reverse.