search

almond-sh / almond

A Scala kernel for Jupyter
https://almond.sh
BSD 3-Clause "New" or "Revised" License
1.59k stars 239 forks source link

Kernel stuck when updating the output in parallel #549

Open gkossakowski opened 4 years ago

gkossakowski commented 4 years ago

I'm running Rainier's model fitting on multiple threads via Scala's parallel collections. I noticed the kernel got stuck burning a CPU indefinitely and lunched VisualVM to see what's going on. In particular, I was interested in whether Rainier went into an infinite loop but it looks like it's a problem with Almond.

Here's what sampler is showing me:

image image

This suggests some race condition in accessing Almond's output reporter.

The thread dump is:


2020-04-26 12:07:45
Full thread dump OpenJDK 64-Bit Server VM (13.0.2+8 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007fb62dcb1ae0, length=39, elements={
0x00007fb65e00c000, 0x00007fb65e014800, 0x00007fb65e017800, 0x00007fb61e008800,
0x00007fb65e814000, 0x00007fb65e015800, 0x00007fb63e01c000, 0x00007fb62e00c800,
0x00007fb61e00b800, 0x00007fb63e1c1800, 0x00007fb61e2d8000, 0x00007fb61e349000,
0x00007fb5df885000, 0x00007fb61e3a2000, 0x00007fb61e3a3000, 0x00007fb61e3a8000,
0x00007fb65e580000, 0x00007fb63e1bf000, 0x00007fb5df895000, 0x00007fb5df89b800,
0x00007fb65e583000, 0x00007fb5df8c7800, 0x00007fb64e289000, 0x00007fb63e1c0000,
0x00007fb60e6fb800, 0x00007fb57af3f800, 0x00007fb5dfd9c800, 0x00007fb5be193800,
0x00007fb65e7fb000, 0x00007fb58a942800, 0x00007fb63e4ad800, 0x00007fb58a943800,
0x00007fb5df0ea000, 0x00007fb58aa0d800, 0x00007fb56e8ee000, 0x00007fb65e657000,
0x00007fb58aa0e000, 0x00007fb5dfe83000, 0x00007fb58aa1d000
}

"main" #1 prio=5 os_prio=31 cpu=925.56ms elapsed=163599.57s tid=0x00007fb65e00c000 nid=0x1703 waiting on condition  [0x0000700005f45000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x00000006000d8600> (a cats.effect.internals.IOPlatform$OneShotLatch)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@13.0.2/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@13.0.2/AbstractQueuedSynchronizer.java:1039)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@13.0.2/AbstractQueuedSynchronizer.java:1345)
        at cats.effect.internals.IOPlatform$.$anonfun$unsafeResync$2(IOPlatform.scala:51)
        at cats.effect.internals.IOPlatform$$$Lambda$398/0x0000000800ee6040.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
        at scala.concurrent.package$.blocking(package.scala:146)
        at cats.effect.internals.IOPlatform$.unsafeResync(IOPlatform.scala:51)
        at cats.effect.IO.unsafeRunTimed(IO.scala:324)
        at cats.effect.IO.unsafeRunSync(IO.scala:239)
        at almond.ScalaKernel$.run(ScalaKernel.scala:151)
        at almond.ScalaKernel$.run(ScalaKernel.scala:15)
        at caseapp.core.app.CaseApp.$anonfun$main$2(CaseApp.scala:68)
        at caseapp.core.app.CaseApp.$anonfun$main$2$adapted(CaseApp.scala:68)
        at caseapp.core.app.CaseApp$$Lambda$103/0x0000000800c4e840.apply(Unknown Source)
        at scala.util.Either.fold(Either.scala:191)
        at caseapp.core.app.CaseApp.main(CaseApp.scala:68)
        at almond.ScalaKernel.main(ScalaKernel.scala)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@13.0.2/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@13.0.2/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@13.0.2/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@13.0.2/Method.java:567)
        at coursier.bootstrap.launcher.a.a(Unknown Source)
        at coursier.bootstrap.launcher.Launcher.main(Unknown Source)

   Locked ownable synchronizers:
        - None

"Reference Handler" #2 daemon prio=10 os_prio=31 cpu=13.29ms elapsed=163599.55s tid=0x00007fb65e014800 nid=0x4903 waiting on condition  [0x000070000665b000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.ref.Reference.waitForReferencePendingList(java.base@13.0.2/Native Method)
        at java.lang.ref.Reference.processPendingReferences(java.base@13.0.2/Reference.java:241)
        at java.lang.ref.Reference$ReferenceHandler.run(java.base@13.0.2/Reference.java:213)

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=31 cpu=1.10ms elapsed=163599.55s tid=0x00007fb65e017800 nid=0x3b03 in Object.wait()  [0x000070000675e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@13.0.2/Native Method)
        - waiting on <no object reference available>
        at java.lang.ref.ReferenceQueue.remove(java.base@13.0.2/ReferenceQueue.java:155)
        - locked <0x0000000601c01e00> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(java.base@13.0.2/ReferenceQueue.java:176)
        at java.lang.ref.Finalizer$FinalizerThread.run(java.base@13.0.2/Finalizer.java:170)

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=0.29ms elapsed=163599.55s tid=0x00007fb61e008800 nid=0x5503 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 cpu=50130.06ms elapsed=163599.55s tid=0x00007fb65e814000 nid=0x5703 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

   Locked ownable synchronizers:
        - None

"C1 CompilerThread0" #13 daemon prio=9 os_prio=31 cpu=8226.43ms elapsed=163599.55s tid=0x00007fb65e015800 nid=0xa603 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

   Locked ownable synchronizers:
        - None

"Sweeper thread" #17 daemon prio=9 os_prio=31 cpu=5369.25ms elapsed=163599.55s tid=0x00007fb63e01c000 nid=0xa403 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Service Thread" #18 daemon prio=9 os_prio=31 cpu=24.54ms elapsed=163599.54s tid=0x00007fb62e00c800 nid=0x5b03 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Common-Cleaner" #19 daemon prio=8 os_prio=31 cpu=83.99ms elapsed=163599.54s tid=0x00007fb61e00b800 nid=0x5e03 in Object.wait()  [0x000070000707c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(java.base@13.0.2/Native Method)
        - waiting on <no object reference available>
        at java.lang.ref.ReferenceQueue.remove(java.base@13.0.2/ReferenceQueue.java:155)
        - locked <0x0000000601c01910> (a java.lang.ref.ReferenceQueue$Lock)
        at jdk.internal.ref.CleanerImpl.run(java.base@13.0.2/CleanerImpl.java:148)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)
        at jdk.internal.misc.InnocuousThread.run(java.base@13.0.2/InnocuousThread.java:134)

   Locked ownable synchronizers:
        - None

"scala-kernel-zeromq-open-close-1" #21 daemon prio=5 os_prio=31 cpu=51.73ms elapsed=163598.38s tid=0x00007fb63e1c1800 nid=0x8f07 waiting on condition  [0x0000700007ea6000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x00000006021abe20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@13.0.2/AbstractQueuedSynchronizer.java:2081)
        at java.util.concurrent.LinkedBlockingQueue.take(java.base@13.0.2/LinkedBlockingQueue.java:433)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@13.0.2/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"reaper-1" #22 daemon prio=5 os_prio=31 cpu=7.70ms elapsed=163598.34s tid=0x00007fb61e2d8000 nid=0x7203 runnable  [0x00007000082b2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueue.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@13.0.2/KQueueSelectorImpl.java:122)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@13.0.2/SelectorImpl.java:124)
        - locked <0x00000006002001f0> (a sun.nio.ch.Util$2)
        - locked <0x0000000600200190> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(java.base@13.0.2/SelectorImpl.java:136)
        at zmq.poll.Poller.run(Poller.java:234)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"iothread-2" #23 daemon prio=5 os_prio=31 cpu=11.08ms elapsed=163598.34s tid=0x00007fb61e349000 nid=0x7303 runnable  [0x00007000083b5000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueue.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@13.0.2/KQueueSelectorImpl.java:122)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@13.0.2/SelectorImpl.java:124)
        - locked <0x0000000600200480> (a sun.nio.ch.Util$2)
        - locked <0x0000000600200420> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(java.base@13.0.2/SelectorImpl.java:136)
        at zmq.poll.Poller.run(Poller.java:234)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"iothread-3" #24 daemon prio=5 os_prio=31 cpu=2381.09ms elapsed=163598.34s tid=0x00007fb5df885000 nid=0x7403 runnable  [0x00007000084b8000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueue.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@13.0.2/KQueueSelectorImpl.java:122)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@13.0.2/SelectorImpl.java:124)
        - locked <0x0000000600200698> (a sun.nio.ch.Util$2)
        - locked <0x0000000600200638> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(java.base@13.0.2/SelectorImpl.java:136)
        at zmq.poll.Poller.run(Poller.java:234)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"iothread-4" #25 daemon prio=5 os_prio=31 cpu=2514.14ms elapsed=163598.34s tid=0x00007fb61e3a2000 nid=0x7603 runnable  [0x00007000085bb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueue.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@13.0.2/KQueueSelectorImpl.java:122)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@13.0.2/SelectorImpl.java:124)
        - locked <0x00000006002008b0> (a sun.nio.ch.Util$2)
        - locked <0x0000000600200850> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(java.base@13.0.2/SelectorImpl.java:136)
        at zmq.poll.Poller.run(Poller.java:234)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"iothread-5" #26 daemon prio=5 os_prio=31 cpu=2.79ms elapsed=163598.33s tid=0x00007fb61e3a3000 nid=0x7903 runnable  [0x00007000086be000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueue.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@13.0.2/KQueueSelectorImpl.java:122)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@13.0.2/SelectorImpl.java:124)
        - locked <0x0000000600200ac8> (a sun.nio.ch.Util$2)
        - locked <0x0000000600200a68> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(java.base@13.0.2/SelectorImpl.java:136)
        at zmq.poll.Poller.run(Poller.java:234)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"scala-kernel-zeromq-Requests-1" #28 daemon prio=5 os_prio=31 cpu=89.59ms elapsed=163598.32s tid=0x00007fb61e3a8000 nid=0x8303 waiting on condition  [0x00007000087c1000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x00000006021a0d68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@13.0.2/AbstractQueuedSynchronizer.java:2081)
        at java.util.concurrent.LinkedBlockingQueue.take(java.base@13.0.2/LinkedBlockingQueue.java:433)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@13.0.2/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"scala-kernel-zeromq-Control-1" #29 daemon prio=5 os_prio=31 cpu=0.45ms elapsed=163598.31s tid=0x00007fb65e580000 nid=0x7c03 waiting on condition  [0x00007000088c4000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x00000006021b75d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@13.0.2/AbstractQueuedSynchronizer.java:2081)
        at java.util.concurrent.LinkedBlockingQueue.take(java.base@13.0.2/LinkedBlockingQueue.java:433)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@13.0.2/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"scala-kernel-zeromq-Publish-1" #30 daemon prio=5 os_prio=31 cpu=1287922.57ms elapsed=163598.31s tid=0x00007fb63e1bf000 nid=0x7e03 runnable  [0x00007000089c7000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileDispatcherImpl.read0(java.base@13.0.2/Native Method)
        at sun.nio.ch.FileDispatcherImpl.read(java.base@13.0.2/FileDispatcherImpl.java:48)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(java.base@13.0.2/IOUtil.java:276)
        at sun.nio.ch.IOUtil.read(java.base@13.0.2/IOUtil.java:245)
        at sun.nio.ch.IOUtil.read(java.base@13.0.2/IOUtil.java:223)
        at sun.nio.ch.SourceChannelImpl.read(java.base@13.0.2/SourceChannelImpl.java:277)
        at zmq.Signaler.recv(Signaler.java:165)
        at zmq.Mailbox.recv(Mailbox.java:97)
        at zmq.SocketBase.processCommands(SocketBase.java:937)
        at zmq.SocketBase.send(SocketBase.java:702)
        at org.zeromq.ZMQ$Socket.send(ZMQ.java:3192)
        at almond.channels.zeromq.ZeromqSocketImpl.$anonfun$send$7(ZeromqSocketImpl.scala:115)
        at almond.channels.zeromq.ZeromqSocketImpl.$anonfun$send$7$adapted(ZeromqSocketImpl.scala:114)
        at almond.channels.zeromq.ZeromqSocketImpl$$Lambda$681/0x0000000800fe5040.apply(Unknown Source)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at almond.channels.zeromq.ZeromqSocketImpl.$anonfun$send$2(ZeromqSocketImpl.scala:114)
        at almond.channels.zeromq.ZeromqSocketImpl$$Lambda$680/0x0000000800fe4440.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
        at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
        at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
        at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
        at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - <0x00000006002101f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        - <0x0000000600225d00> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"scala-kernel-zeromq-Input-1" #31 daemon prio=5 os_prio=31 cpu=0.63ms elapsed=163598.31s tid=0x00007fb5df895000 nid=0xaa03 waiting on condition  [0x0000700008aca000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x00000006021abed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@13.0.2/AbstractQueuedSynchronizer.java:2081)
        at java.util.concurrent.LinkedBlockingQueue.take(java.base@13.0.2/LinkedBlockingQueue.java:433)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@13.0.2/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"scala-kernel-zeromq-open-close-2" #32 daemon prio=5 os_prio=31 cpu=106.40ms elapsed=163598.30s tid=0x00007fb5df89b800 nid=0xac03 waiting on condition  [0x0000700008bcd000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x00000006021abe20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@13.0.2/AbstractQueuedSynchronizer.java:2081)
        at java.util.concurrent.LinkedBlockingQueue.take(java.base@13.0.2/LinkedBlockingQueue.java:433)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@13.0.2/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"ZeroMQ-HeartBeat" #27 daemon prio=5 os_prio=31 cpu=1.31ms elapsed=163598.30s tid=0x00007fb65e583000 nid=0x15103 runnable  [0x0000700008cd0000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueue.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@13.0.2/KQueueSelectorImpl.java:122)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@13.0.2/SelectorImpl.java:124)
        - locked <0x0000000600226288> (a sun.nio.ch.Util$2)
        - locked <0x0000000600226228> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(java.base@13.0.2/SelectorImpl.java:136)
        at zmq.ZMQ.poll(ZMQ.java:711)
        at zmq.ZMQ.poll(ZMQ.java:619)
        at zmq.Proxy.start(Proxy.java:64)
        at zmq.Proxy.proxy(Proxy.java:17)
        at zmq.ZMQ.proxy(ZMQ.java:787)
        at org.zeromq.ZMQ.proxy(ZMQ.java:333)
        at almond.channels.zeromq.ZeromqConnection$$anon$1.run(ZeromqConnection.scala:109)

   Locked ownable synchronizers:
        - None

"scala-kernel-dummy-stuff-1" #33 daemon prio=5 os_prio=31 cpu=7327.79ms elapsed=163598.19s tid=0x00007fb5df8c7800 nid=0x14f03 waiting on condition  [0x0000700008dd3000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x00000006021b8138> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@13.0.2/AbstractQueuedSynchronizer.java:2081)
        at java.util.concurrent.LinkedBlockingQueue.take(java.base@13.0.2/LinkedBlockingQueue.java:433)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@13.0.2/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"scala-interpreter-1" #34 daemon prio=5 os_prio=31 cpu=125889.35ms elapsed=163598.15s tid=0x00007fb64e289000 nid=0xaf03 in Object.wait()  [0x0000700008ed3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@13.0.2/Native Method)
        - waiting on <no object reference available>
        at java.util.concurrent.ForkJoinTask.externalAwaitDone(java.base@13.0.2/ForkJoinTask.java:330)
        - locked <0x0000000612283f00> (a scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask)
        at java.util.concurrent.ForkJoinTask.doJoin(java.base@13.0.2/ForkJoinTask.java:398)
        at java.util.concurrent.ForkJoinTask.join(java.base@13.0.2/ForkJoinTask.java:721)
        at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379)
        at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440)
        at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:423)
        at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:416)
        at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:60)
        at scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult(Tasks.scala:555)
        at scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult$(Tasks.scala:555)
        at scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:84)
        at scala.collection.parallel.ParIterableLike.map(ParIterableLike.scala:501)
        at scala.collection.parallel.ParIterableLike.map$(ParIterableLike.scala:500)
        at scala.collection.parallel.immutable.ParHashSet.map(ParHashSet.scala:50)
        at ammonite.$sess.cmd47$Helper.$anonfun$countryStats$2(cmd47.sc:4)
        at ammonite.$sess.cmd47$Helper$$Lambda$3778/0x0000000801bd1840.apply(Unknown Source)
        at ammonite.$sess.cmd38$Helper.time(cmd38.sc:3)
        at ammonite.$sess.cmd47$Helper.<init>(cmd47.sc:12)
        at ammonite.$sess.cmd47$.<init>(cmd47.sc:7)
        at ammonite.$sess.cmd47$.<clinit>(cmd47.sc)
        at ammonite.$sess.cmd47.$main(cmd47.sc)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@13.0.2/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@13.0.2/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@13.0.2/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@13.0.2/Method.java:567)
        at ammonite.runtime.Evaluator$$anon$1.$anonfun$evalMain$1(Evaluator.scala:108)
        at ammonite.runtime.Evaluator$$anon$1$$Lambda$334/0x0000000800e71c40.apply(Unknown Source)
        at ammonite.util.Util$.withContextClassloader(Util.scala:24)
        at ammonite.runtime.Evaluator$$anon$1.evalMain(Evaluator.scala:90)
        at ammonite.runtime.Evaluator$$anon$1.$anonfun$processLine$2(Evaluator.scala:127)
        at ammonite.runtime.Evaluator$$anon$1$$Lambda$2394/0x0000000801620840.apply(Unknown Source)
        at ammonite.util.Catching.map(Res.scala:117)
        at ammonite.runtime.Evaluator$$anon$1.$anonfun$processLine$1(Evaluator.scala:121)
        at ammonite.runtime.Evaluator$$anon$1$$Lambda$2393/0x000000080161f840.apply(Unknown Source)
        at ammonite.util.Res$Success.flatMap(Res.scala:62)
        at ammonite.runtime.Evaluator$$anon$1.processLine(Evaluator.scala:120)
        at ammonite.interp.Interpreter.$anonfun$evaluateLine$4(Interpreter.scala:296)
        at ammonite.interp.Interpreter$$Lambda$2391/0x000000080161e840.apply(Unknown Source)
        at ammonite.util.Res$Success.flatMap(Res.scala:62)
        at ammonite.interp.Interpreter.$anonfun$evaluateLine$2(Interpreter.scala:282)
        at ammonite.interp.Interpreter$$Lambda$1877/0x0000000801461840.apply(Unknown Source)
        at ammonite.util.Catching.flatMap(Res.scala:115)
        at ammonite.interp.Interpreter.evaluateLine(Interpreter.scala:281)
        - locked <0x00000006021d7e90> (a almond.amm.AmmInterpreter$$anon$1)
        at ammonite.interp.Interpreter.$anonfun$processLine$6(Interpreter.scala:268)
        at ammonite.interp.Interpreter$$Lambda$1876/0x0000000801461040.apply(Unknown Source)
        at ammonite.util.Res$Success.flatMap(Res.scala:62)
        at ammonite.interp.Interpreter.$anonfun$processLine$4(Interpreter.scala:252)
        at ammonite.interp.Interpreter$$Lambda$1570/0x00000008011eb040.apply(Unknown Source)
        at ammonite.util.Res$Success.flatMap(Res.scala:62)
        at ammonite.interp.Interpreter.$anonfun$processLine$2(Interpreter.scala:245)
        at ammonite.interp.Interpreter$$Lambda$966/0x00000008010af840.apply(Unknown Source)
        at ammonite.util.Catching.flatMap(Res.scala:115)
        at ammonite.interp.Interpreter.processLine(Interpreter.scala:244)
        - locked <0x00000006021d7e90> (a almond.amm.AmmInterpreter$$anon$1)
        at almond.Execute.$anonfun$ammResult$9(Execute.scala:227)
        at almond.Execute$$Lambda$950/0x000000080109f840.apply(Unknown Source)
        at almond.internals.CaptureImpl.$anonfun$apply$2(CaptureImpl.scala:53)
        at almond.internals.CaptureImpl$$Lambda$954/0x00000008010a9040.apply(Unknown Source)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at scala.Console$.withErr(Console.scala:196)
        at almond.internals.CaptureImpl.$anonfun$apply$1(CaptureImpl.scala:45)
        at almond.internals.CaptureImpl$$Lambda$953/0x00000008010a8440.apply(Unknown Source)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at scala.Console$.withOut(Console.scala:167)
        at almond.internals.CaptureImpl.apply(CaptureImpl.scala:45)
        at almond.Execute.capturingOutput(Execute.scala:165)
        at almond.Execute.$anonfun$ammResult$8(Execute.scala:223)
        at almond.Execute$$Lambda$948/0x000000080109e840.apply(Unknown Source)
        at almond.Execute.$anonfun$withClientStdin$1(Execute.scala:145)
        at almond.Execute$$Lambda$949/0x000000080109ec40.apply(Unknown Source)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at scala.Console$.withIn(Console.scala:230)
        at almond.Execute.withClientStdin(Execute.scala:141)
        at almond.Execute.$anonfun$ammResult$7(Execute.scala:223)
        at almond.Execute$$Lambda$947/0x000000080109e440.apply(Unknown Source)
        at almond.Execute.withInputManager(Execute.scala:133)
        at almond.Execute.$anonfun$ammResult$6(Execute.scala:222)
        at almond.Execute$$Lambda$944/0x000000080109d840.apply(Unknown Source)
        at ammonite.repl.Signaller.apply(Signaller.scala:28)
        at almond.Execute.interruptible(Execute.scala:182)
        at almond.Execute.$anonfun$ammResult$5(Execute.scala:221)
        at almond.Execute$$Lambda$943/0x000000080109d040.apply(Unknown Source)
        at ammonite.util.Res$Success.flatMap(Res.scala:62)
        at almond.Execute.$anonfun$ammResult$1(Execute.scala:212)
        at almond.Execute$$Lambda$872/0x0000000801072840.apply(Unknown Source)
        at almond.Execute.withOutputHandler(Execute.scala:156)
        at almond.Execute.ammResult(Execute.scala:212)
        at almond.Execute.apply(Execute.scala:296)
        at almond.ScalaInterpreter.execute(ScalaInterpreter.scala:120)
        at almond.interpreter.InterpreterToIOInterpreter.$anonfun$execute$2(InterpreterToIOInterpreter.scala:69)
        at almond.interpreter.InterpreterToIOInterpreter$$Lambda$870/0x0000000801071840.apply(Unknown Source)
        at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
        at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
        at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
        at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
        at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - <0x000000060022ca40> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"scala-kernel-zeromq-polling-1" #35 daemon prio=5 os_prio=31 cpu=8419.51ms elapsed=163598.13s tid=0x00007fb63e1c0000 nid=0xb203 runnable  [0x0000700008fd9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueue.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(java.base@13.0.2/KQueueSelectorImpl.java:122)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@13.0.2/SelectorImpl.java:124)
        - locked <0x000000060021fb60> (a sun.nio.ch.Util$2)
        - locked <0x000000060020a230> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(java.base@13.0.2/SelectorImpl.java:136)
        at zmq.ZMQ.poll(ZMQ.java:717)
        at org.zeromq.ZMQ.poll(ZMQ.java:357)
        at org.zeromq.ZMQ.poll(ZMQ.java:347)
        at almond.channels.zeromq.ZeromqConnection.$anonfun$tryRead$3(ZeromqConnection.scala:186)
        at almond.channels.zeromq.ZeromqConnection.$anonfun$tryRead$3$adapted(ZeromqConnection.scala:185)
        at almond.channels.zeromq.ZeromqConnection$$Lambda$761/0x0000000801024840.apply(Unknown Source)
        at almond.channels.zeromq.ZeromqConnection.withSelector(ZeromqConnection.scala:136)
        at almond.channels.zeromq.ZeromqConnection.$anonfun$tryRead$1(ZeromqConnection.scala:185)
        at almond.channels.zeromq.ZeromqConnection$$Lambda$448/0x0000000800f1f840.apply(Unknown Source)
        at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:87)
        at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:355)
        at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:376)
        at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:316)
        at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - <0x0000000600274a58> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"AppKit Thread" #49 daemon prio=5 os_prio=31 cpu=873.22ms elapsed=163567.33s tid=0x00007fb60e6fb800 nid=0x307 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Java2D Queue Flusher" #51 daemon prio=10 os_prio=31 cpu=23565.39ms elapsed=163567.20s tid=0x00007fb57af3f800 nid=0x17307 in Object.wait()  [0x000070000ae48000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(java.base@13.0.2/Native Method)
        - waiting on <no object reference available>
        at sun.java2d.opengl.OGLRenderQueue$QueueFlusher.run(java.desktop@13.0.2/OGLRenderQueue.java:205)
        - locked <0x00000006048e2d70> (a sun.java2d.opengl.OGLRenderQueue$QueueFlusher)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"Java2D Disposer" #52 daemon prio=10 os_prio=31 cpu=0.68ms elapsed=163567.15s tid=0x00007fb5dfd9c800 nid=0xc41f in Object.wait()  [0x00007000097f1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@13.0.2/Native Method)
        - waiting on <no object reference available>
        at java.lang.ref.ReferenceQueue.remove(java.base@13.0.2/ReferenceQueue.java:155)
        - locked <0x00000006048ceea0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(java.base@13.0.2/ReferenceQueue.java:176)
        at sun.java2d.Disposer.run(java.desktop@13.0.2/Disposer.java:144)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"scala-execution-context-global-92" #92 daemon prio=5 os_prio=31 cpu=126165.31ms elapsed=158948.55s tid=0x00007fb5be193800 nid=0x405f in Object.wait()  [0x00007000098f3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@13.0.2/Native Method)
        - waiting on <no object reference available>
        at java.util.concurrent.ForkJoinTask.internalWait(java.base@13.0.2/ForkJoinTask.java:311)
        - locked <0x0000000612427d90> (a scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask)
        at java.util.concurrent.ForkJoinPool.awaitJoin(java.base@13.0.2/ForkJoinPool.java:1739)
        at java.util.concurrent.ForkJoinTask.doJoin(java.base@13.0.2/ForkJoinTask.java:397)
        at java.util.concurrent.ForkJoinTask.join(java.base@13.0.2/ForkJoinTask.java:721)
        at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379)
        at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:174)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:157)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:440)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:150)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
        at java.util.concurrent.RecursiveAction.exec(java.base@13.0.2/RecursiveAction.java:189)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@13.0.2/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinTask.doJoin(java.base@13.0.2/ForkJoinTask.java:396)
        at java.util.concurrent.ForkJoinTask.join(java.base@13.0.2/ForkJoinTask.java:721)
        at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379)
        at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440)
        at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult(Tasks.scala:423)
        at scala.collection.parallel.ForkJoinTasks.executeAndWaitResult$(Tasks.scala:416)
        at scala.collection.parallel.ForkJoinTaskSupport.executeAndWaitResult(TaskSupport.scala:60)
        at scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult(Tasks.scala:555)
        at scala.collection.parallel.ExecutionContextTasks.executeAndWaitResult$(Tasks.scala:555)
        at scala.collection.parallel.ExecutionContextTaskSupport.executeAndWaitResult(TaskSupport.scala:84)
        at scala.collection.parallel.ParIterableLike$ResultMapping.leaf(ParIterableLike.scala:960)
        at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
        at scala.collection.parallel.Task$$Lambda$3668/0x0000000801a6e040.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
        at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
        at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
        at scala.collection.parallel.ParIterableLike$ResultMapping.tryLeaf(ParIterableLike.scala:955)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:153)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
        at java.util.concurrent.RecursiveAction.exec(java.base@13.0.2/RecursiveAction.java:189)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@13.0.2/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@13.0.2/ForkJoinPool.java:1016)
        at java.util.concurrent.ForkJoinPool.scan(java.base@13.0.2/ForkJoinPool.java:1665)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@13.0.2/ForkJoinPool.java:1598)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@13.0.2/ForkJoinWorkerThread.java:177)

   Locked ownable synchronizers:
        - None

"scala-execution-context-global-487" #487 daemon prio=5 os_prio=31 cpu=87121.72ms elapsed=1415.85s tid=0x00007fb65e7fb000 nid=0x183e7 waiting on condition  [0x0000700009bfd000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x00000006028c8180> (a java.util.concurrent.ForkJoinPool)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@13.0.2/ForkJoinPool.java:1633)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@13.0.2/ForkJoinWorkerThread.java:177)

   Locked ownable synchronizers:
        - None

"scala-execution-context-global-489" #489 daemon prio=5 os_prio=31 cpu=123171.46ms elapsed=1415.85s tid=0x00007fb58a942800 nid=0x1c84f waiting on condition  [0x000070000a820000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x0000000620cafd70> (a cats.effect.internals.IOPlatform$OneShotLatch)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@13.0.2/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@13.0.2/AbstractQueuedSynchronizer.java:1039)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@13.0.2/AbstractQueuedSynchronizer.java:1345)
        at cats.effect.internals.IOPlatform$.$anonfun$unsafeResync$2(IOPlatform.scala:51)
        at cats.effect.internals.IOPlatform$$$Lambda$398/0x0000000800ee6040.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base@13.0.2/ForkJoinPool.java:3127)
        at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87)
        at scala.concurrent.package$.blocking(package.scala:146)
        at cats.effect.internals.IOPlatform$.unsafeResync(IOPlatform.scala:51)
        at cats.effect.IO.unsafeRunTimed(IO.scala:324)
        at cats.effect.IO.unsafeRunSync(IO.scala:239)
        at almond.interpreter.comm.DefaultCommHandler.publish(DefaultCommHandler.scala:47)
        at almond.interpreter.comm.DefaultCommHandler.updateDisplay(DefaultCommHandler.scala:70)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.$anonfun$updateDisplay$1(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.$anonfun$updateDisplay$1$adapted(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler$$Lambda$3409/0x0000000801979040.apply(Unknown Source)
        at scala.Option.foreach(Option.scala:407)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.updateDisplay(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.api.OutputHandler$StableOutputHandler.updateDisplay(OutputHandler.scala:84)
        at almond.interpreter.api.OutputHandler$UpdateHelpers.updateHtml(OutputHandler.scala:57)
        at almond.interpreter.api.OutputHandler$UpdateHelpers.updateHtml$(OutputHandler.scala:56)
        at almond.interpreter.api.OutputHandler.updateHtml(OutputHandler.scala:16)
        at com.stripe.rainier.notebook.HTMLProgress.refresh(HTMLProgress.scala:24)
        at com.stripe.rainier.sampler.Driver$.collectSamples(Driver.scala:112)
        at com.stripe.rainier.sampler.Driver$.sample(Driver.scala:33)
        at com.stripe.rainier.core.Model.$anonfun$sample$1(Model.scala:20)
        at com.stripe.rainier.core.Model.$anonfun$sample$1$adapted(Model.scala:19)
        at com.stripe.rainier.core.Model$$Lambda$3310/0x000000080191d040.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:290)
        at com.stripe.rainier.core.Model.sample(Model.scala:19)
        at ammonite.$sess.cmd43$Helper.$anonfun$predictRt$6(cmd43.sc:25)
        at ammonite.$sess.cmd43$Helper$$Lambda$3765/0x0000000801bbac40.apply(Unknown Source)
        at ammonite.$sess.cmd42$Helper.retry(cmd42.sc:5)
        at ammonite.$sess.cmd43$Helper.predictRt(cmd43.sc:25)
        at ammonite.$sess.cmd47$Helper.$anonfun$countryStats$3(cmd47.sc:8)
        at ammonite.$sess.cmd47$Helper$$Lambda$3779/0x0000000801bd2040.apply(Unknown Source)
        at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116)
        at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113)
        at scala.collection.parallel.immutable.ParHashSet$ParHashSetIterator.map2combiner(ParHashSet.scala:81)
        at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1056)
        at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
        at scala.collection.parallel.Task$$Lambda$3668/0x0000000801a6e040.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
        at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
        at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
        at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1053)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:170)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:157)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:440)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:150)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
        at java.util.concurrent.RecursiveAction.exec(java.base@13.0.2/RecursiveAction.java:189)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@13.0.2/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@13.0.2/ForkJoinPool.java:1016)
        at java.util.concurrent.ForkJoinPool.scan(java.base@13.0.2/ForkJoinPool.java:1665)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@13.0.2/ForkJoinPool.java:1598)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@13.0.2/ForkJoinWorkerThread.java:177)

   Locked ownable synchronizers:
        - None

"scala-execution-context-global-490" #490 daemon prio=5 os_prio=31 cpu=105205.34ms elapsed=1415.85s tid=0x00007fb63e4ad800 nid=0x1c567 in Object.wait()  [0x000070000a924000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@13.0.2/Native Method)
        - waiting on <no object reference available>
        at java.util.concurrent.ForkJoinTask.internalWait(java.base@13.0.2/ForkJoinTask.java:311)
        - locked <0x000000061251d318> (a scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask)
        at java.util.concurrent.ForkJoinPool.awaitJoin(java.base@13.0.2/ForkJoinPool.java:1739)
        at java.util.concurrent.ForkJoinTask.doJoin(java.base@13.0.2/ForkJoinTask.java:397)
        at java.util.concurrent.ForkJoinTask.join(java.base@13.0.2/ForkJoinTask.java:721)
        at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync(Tasks.scala:379)
        at scala.collection.parallel.ForkJoinTasks$WrappedTask.sync$(Tasks.scala:379)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.sync(Tasks.scala:440)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:174)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:157)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:440)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:150)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
        at java.util.concurrent.RecursiveAction.exec(java.base@13.0.2/RecursiveAction.java:189)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@13.0.2/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@13.0.2/ForkJoinPool.java:1016)
        at java.util.concurrent.ForkJoinPool.scan(java.base@13.0.2/ForkJoinPool.java:1665)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@13.0.2/ForkJoinPool.java:1598)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@13.0.2/ForkJoinWorkerThread.java:177)

   Locked ownable synchronizers:
        - None

"scala-execution-context-global-497" #497 daemon prio=5 os_prio=31 cpu=123072.60ms elapsed=1415.85s tid=0x00007fb58a943800 nid=0x1c9ab waiting on condition  [0x000070000af4a000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x0000000620f07c28> (a cats.effect.internals.IOPlatform$OneShotLatch)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@13.0.2/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@13.0.2/AbstractQueuedSynchronizer.java:1039)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@13.0.2/AbstractQueuedSynchronizer.java:1345)
        at cats.effect.internals.IOPlatform$.$anonfun$unsafeResync$2(IOPlatform.scala:51)
        at cats.effect.internals.IOPlatform$$$Lambda$398/0x0000000800ee6040.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base@13.0.2/ForkJoinPool.java:3127)
        at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87)
        at scala.concurrent.package$.blocking(package.scala:146)
        at cats.effect.internals.IOPlatform$.unsafeResync(IOPlatform.scala:51)
        at cats.effect.IO.unsafeRunTimed(IO.scala:324)
        at cats.effect.IO.unsafeRunSync(IO.scala:239)
        at almond.interpreter.comm.DefaultCommHandler.publish(DefaultCommHandler.scala:47)
        at almond.interpreter.comm.DefaultCommHandler.updateDisplay(DefaultCommHandler.scala:70)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.$anonfun$updateDisplay$1(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.$anonfun$updateDisplay$1$adapted(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler$$Lambda$3409/0x0000000801979040.apply(Unknown Source)
        at scala.Option.foreach(Option.scala:407)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.updateDisplay(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.api.OutputHandler$StableOutputHandler.updateDisplay(OutputHandler.scala:84)
        at almond.interpreter.api.OutputHandler$UpdateHelpers.updateHtml(OutputHandler.scala:57)
        at almond.interpreter.api.OutputHandler$UpdateHelpers.updateHtml$(OutputHandler.scala:56)
        at almond.interpreter.api.OutputHandler.updateHtml(OutputHandler.scala:16)
        at com.stripe.rainier.notebook.HTMLProgress.refresh(HTMLProgress.scala:24)
        at com.stripe.rainier.sampler.Driver$.warmup(Driver.scala:83)
        at com.stripe.rainier.sampler.Driver$.sample(Driver.scala:23)
        at com.stripe.rainier.core.Model.$anonfun$sample$1(Model.scala:20)
        at com.stripe.rainier.core.Model.$anonfun$sample$1$adapted(Model.scala:19)
        at com.stripe.rainier.core.Model$$Lambda$3310/0x000000080191d040.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:290)
        at com.stripe.rainier.core.Model.sample(Model.scala:19)
        at ammonite.$sess.cmd43$Helper.$anonfun$predictRt$6(cmd43.sc:25)
        at ammonite.$sess.cmd43$Helper$$Lambda$3765/0x0000000801bbac40.apply(Unknown Source)
        at ammonite.$sess.cmd42$Helper.retry(cmd42.sc:5)
        at ammonite.$sess.cmd43$Helper.predictRt(cmd43.sc:25)
        at ammonite.$sess.cmd47$Helper.$anonfun$countryStats$3(cmd47.sc:8)
        at ammonite.$sess.cmd47$Helper$$Lambda$3779/0x0000000801bd2040.apply(Unknown Source)
        at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116)
        at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113)
        at scala.collection.parallel.immutable.ParHashSet$ParHashSetIterator.map2combiner(ParHashSet.scala:81)
        at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1056)
        at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
        at scala.collection.parallel.Task$$Lambda$3668/0x0000000801a6e040.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
        at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
        at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
        at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1053)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:160)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:157)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:440)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:150)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
        at java.util.concurrent.RecursiveAction.exec(java.base@13.0.2/RecursiveAction.java:189)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@13.0.2/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@13.0.2/ForkJoinPool.java:1016)
        at java.util.concurrent.ForkJoinPool.scan(java.base@13.0.2/ForkJoinPool.java:1665)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@13.0.2/ForkJoinPool.java:1598)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@13.0.2/ForkJoinWorkerThread.java:177)

   Locked ownable synchronizers:
        - None

"scala-execution-context-global-496" #496 daemon prio=5 os_prio=31 cpu=123140.80ms elapsed=1415.85s tid=0x00007fb5df0ea000 nid=0x14b13 waiting on condition  [0x000070000b150000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x0000000620c00028> (a cats.effect.internals.IOPlatform$OneShotLatch)
        at java.util.concurrent.locks.LockSupport.park(java.base@13.0.2/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@13.0.2/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@13.0.2/AbstractQueuedSynchronizer.java:1039)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@13.0.2/AbstractQueuedSynchronizer.java:1345)
        at cats.effect.internals.IOPlatform$.$anonfun$unsafeResync$2(IOPlatform.scala:51)
        at cats.effect.internals.IOPlatform$$$Lambda$398/0x0000000800ee6040.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75)
        at java.util.concurrent.ForkJoinPool.managedBlock(java.base@13.0.2/ForkJoinPool.java:3127)
        at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87)
        at scala.concurrent.package$.blocking(package.scala:146)
        at cats.effect.internals.IOPlatform$.unsafeResync(IOPlatform.scala:51)
        at cats.effect.IO.unsafeRunTimed(IO.scala:324)
        at cats.effect.IO.unsafeRunSync(IO.scala:239)
        at almond.interpreter.comm.DefaultCommHandler.publish(DefaultCommHandler.scala:47)
        at almond.interpreter.comm.DefaultCommHandler.updateDisplay(DefaultCommHandler.scala:70)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.$anonfun$updateDisplay$1(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.$anonfun$updateDisplay$1$adapted(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler$$Lambda$3409/0x0000000801979040.apply(Unknown Source)
        at scala.Option.foreach(Option.scala:407)
        at almond.interpreter.messagehandlers.InterpreterMessageHandlers$QueueOutputHandler.updateDisplay(InterpreterMessageHandlers.scala:255)
        at almond.interpreter.api.OutputHandler$StableOutputHandler.updateDisplay(OutputHandler.scala:84)
        at almond.interpreter.api.OutputHandler$UpdateHelpers.updateHtml(OutputHandler.scala:57)
        at almond.interpreter.api.OutputHandler$UpdateHelpers.updateHtml$(OutputHandler.scala:56)
        at almond.interpreter.api.OutputHandler.updateHtml(OutputHandler.scala:16)
        at com.stripe.rainier.notebook.HTMLProgress.refresh(HTMLProgress.scala:24)
        at com.stripe.rainier.sampler.Driver$.warmup(Driver.scala:83)
        at com.stripe.rainier.sampler.Driver$.sample(Driver.scala:23)
        at com.stripe.rainier.core.Model.$anonfun$sample$1(Model.scala:20)
        at com.stripe.rainier.core.Model.$anonfun$sample$1$adapted(Model.scala:19)
        at com.stripe.rainier.core.Model$$Lambda$3310/0x000000080191d040.apply(Unknown Source)
        at scala.collection.immutable.List.map(List.scala:290)
        at com.stripe.rainier.core.Model.sample(Model.scala:19)
        at ammonite.$sess.cmd43$Helper.$anonfun$predictRt$6(cmd43.sc:25)
        at ammonite.$sess.cmd43$Helper$$Lambda$3765/0x0000000801bbac40.apply(Unknown Source)
        at ammonite.$sess.cmd42$Helper.retry(cmd42.sc:5)
        at ammonite.$sess.cmd43$Helper.predictRt(cmd43.sc:25)
        at ammonite.$sess.cmd47$Helper.$anonfun$countryStats$3(cmd47.sc:8)
        at ammonite.$sess.cmd47$Helper$$Lambda$3779/0x0000000801bd2040.apply(Unknown Source)
        at scala.collection.parallel.AugmentedIterableIterator.map2combiner(RemainsIterator.scala:116)
        at scala.collection.parallel.AugmentedIterableIterator.map2combiner$(RemainsIterator.scala:113)
        at scala.collection.parallel.immutable.ParHashSet$ParHashSetIterator.map2combiner(ParHashSet.scala:81)
        at scala.collection.parallel.ParIterableLike$Map.leaf(ParIterableLike.scala:1056)
        at scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
        at scala.collection.parallel.Task$$Lambda$3668/0x0000000801a6e040.apply$mcV$sp(Unknown Source)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
        at scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
        at scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
        at scala.collection.parallel.ParIterableLike$Map.tryLeaf(ParIterableLike.scala:1053)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:160)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:157)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:440)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:150)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
        at java.util.concurrent.RecursiveAction.exec(java.base@13.0.2/RecursiveAction.java:189)
        at java.util.concurrent.ForkJoinTask.doExec(java.base@13.0.2/ForkJoinTask.java:290)
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@13.0.2/ForkJoinPool.java:1016)
        at java.util.concurrent.ForkJoinPool.scan(java.base@13.0.2/ForkJoinPool.java:1665)
        at java.util.concurrent.ForkJoinPool.runWorker(java.base@13.0.2/ForkJoinPool.java:1598)
        at java.util.concurrent.ForkJoinWorkerThread.run(java.base@13.0.2/ForkJoinWorkerThread.java:177)

   Locked ownable synchronizers:
        - None

"Attach Listener" #506 daemon prio=9 os_prio=31 cpu=121.61ms elapsed=692.17s tid=0x00007fb58aa0d800 nid=0x191a3 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"RMI TCP Accept-0" #510 daemon prio=9 os_prio=31 cpu=2.25ms elapsed=503.02s tid=0x00007fb56e8ee000 nid=0x19713 runnable  [0x00007000081af000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.Net.accept(java.base@13.0.2/Native Method)
        at sun.nio.ch.NioSocketImpl.accept(java.base@13.0.2/NioSocketImpl.java:755)
        at java.net.ServerSocket.implAccept(java.base@13.0.2/ServerSocket.java:662)
        at java.net.ServerSocket.platformImplAccept(java.base@13.0.2/ServerSocket.java:628)
        at java.net.ServerSocket.implAccept(java.base@13.0.2/ServerSocket.java:604)
        at java.net.ServerSocket.implAccept(java.base@13.0.2/ServerSocket.java:561)
        at java.net.ServerSocket.accept(java.base@13.0.2/ServerSocket.java:518)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(jdk.management.agent@13.0.2/LocalRMIServerSocketFactory.java:52)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(java.rmi@13.0.2/TCPTransport.java:394)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(java.rmi@13.0.2/TCPTransport.java:366)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - <0x0000000620fa06a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"RMI Scheduler(0)" #512 daemon prio=9 os_prio=31 cpu=0.76ms elapsed=502.90s tid=0x00007fb65e657000 nid=0xb643 waiting on condition  [0x00007000091df000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@13.0.2/Native Method)
        - parking to wait for  <0x0000000620c58048> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(java.base@13.0.2/LockSupport.java:235)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@13.0.2/AbstractQueuedSynchronizer.java:2123)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@13.0.2/ScheduledThreadPoolExecutor.java:1182)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@13.0.2/ScheduledThreadPoolExecutor.java:899)
        at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@13.0.2/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"RMI TCP Connection(7)-127.0.0.1" #524 daemon prio=9 os_prio=31 cpu=3008.45ms elapsed=175.99s tid=0x00007fb58aa0e000 nid=0x14787 runnable  [0x0000700007202000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.Net.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.NioSocketImpl.park(java.base@13.0.2/NioSocketImpl.java:182)
        at sun.nio.ch.NioSocketImpl.timedRead(java.base@13.0.2/NioSocketImpl.java:286)
        at sun.nio.ch.NioSocketImpl.implRead(java.base@13.0.2/NioSocketImpl.java:310)
        at sun.nio.ch.NioSocketImpl.read(java.base@13.0.2/NioSocketImpl.java:351)
        at sun.nio.ch.NioSocketImpl$1.read(java.base@13.0.2/NioSocketImpl.java:802)
        at java.net.Socket$SocketInputStream.read(java.base@13.0.2/Socket.java:937)
        at java.io.BufferedInputStream.fill(java.base@13.0.2/BufferedInputStream.java:245)
        at java.io.BufferedInputStream.read(java.base@13.0.2/BufferedInputStream.java:264)
        - locked <0x0000000620e5a238> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(java.base@13.0.2/FilterInputStream.java:83)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(java.rmi@13.0.2/TCPTransport.java:544)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(java.rmi@13.0.2/TCPTransport.java:796)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(java.rmi@13.0.2/TCPTransport.java:677)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$3841/0x0000000801c78440.run(java.rmi@13.0.2/Unknown Source)
        at java.security.AccessController.executePrivileged(java.base@13.0.2/AccessController.java:753)
        at java.security.AccessController.doPrivileged(java.base@13.0.2/AccessController.java:391)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(java.rmi@13.0.2/TCPTransport.java:676)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - <0x0000000620e5a650> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        - <0x0000000620e5d150> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"JMX server connection timeout 525" #525 daemon prio=9 os_prio=31 cpu=53.34ms elapsed=175.96s tid=0x00007fb5dfe83000 nid=0xc0af in Object.wait()  [0x00007000090dc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(java.base@13.0.2/Native Method)
        - waiting on <no object reference available>
        at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(java.management@13.0.2/ServerCommunicatorAdmin.java:171)
        - locked <0x0000000620eaff00> (a [I)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - None

"RMI TCP Connection(10)-127.0.0.1" #526 daemon prio=9 os_prio=31 cpu=1892.53ms elapsed=174.89s tid=0x00007fb58aa1d000 nid=0x18f0b runnable  [0x00007000092e2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.Net.poll(java.base@13.0.2/Native Method)
        at sun.nio.ch.NioSocketImpl.park(java.base@13.0.2/NioSocketImpl.java:182)
        at sun.nio.ch.NioSocketImpl.timedRead(java.base@13.0.2/NioSocketImpl.java:286)
        at sun.nio.ch.NioSocketImpl.implRead(java.base@13.0.2/NioSocketImpl.java:310)
        at sun.nio.ch.NioSocketImpl.read(java.base@13.0.2/NioSocketImpl.java:351)
        at sun.nio.ch.NioSocketImpl$1.read(java.base@13.0.2/NioSocketImpl.java:802)
        at java.net.Socket$SocketInputStream.read(java.base@13.0.2/Socket.java:937)
        at java.io.BufferedInputStream.fill(java.base@13.0.2/BufferedInputStream.java:245)
        at java.io.BufferedInputStream.read(java.base@13.0.2/BufferedInputStream.java:264)
        - locked <0x0000000620ee7738> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(java.base@13.0.2/FilterInputStream.java:83)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(java.rmi@13.0.2/TCPTransport.java:544)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(java.rmi@13.0.2/TCPTransport.java:796)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(java.rmi@13.0.2/TCPTransport.java:677)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$3841/0x0000000801c78440.run(java.rmi@13.0.2/Unknown Source)
        at java.security.AccessController.executePrivileged(java.base@13.0.2/AccessController.java:753)
        at java.security.AccessController.doPrivileged(java.base@13.0.2/AccessController.java:391)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(java.rmi@13.0.2/TCPTransport.java:676)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@13.0.2/ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@13.0.2/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(java.base@13.0.2/Thread.java:830)

   Locked ownable synchronizers:
        - <0x0000000620b50ec8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        - <0x0000000620e5a8c0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"VM Thread" os_prio=31 cpu=4077.51ms elapsed=163599.56s tid=0x00007fb63e01b000 nid=0x4c03 runnable  

"GC Thread#0" os_prio=31 cpu=897.08ms elapsed=163599.57s tid=0x00007fb64e039800 nid=0x5303 runnable  

"GC Thread#1" os_prio=31 cpu=894.38ms elapsed=163599.12s tid=0x00007fb65ea26800 nid=0x6203 runnable  

"GC Thread#2" os_prio=31 cpu=916.93ms elapsed=163599.12s tid=0x00007fb65ef7f800 nid=0x9a03 runnable  

"GC Thread#3" os_prio=31 cpu=893.85ms elapsed=163599.12s tid=0x00007fb65ea2b000 nid=0x9903 runnable  

"GC Thread#4" os_prio=31 cpu=920.12ms elapsed=163599.12s tid=0x00007fb60e07e000 nid=0x6603 runnable  

"GC Thread#5" os_prio=31 cpu=906.65ms elapsed=163599.12s tid=0x00007fb64e228800 nid=0x9803 runnable  

"GC Thread#6" os_prio=31 cpu=926.39ms elapsed=163599.12s tid=0x00007fb65e34a800 nid=0x6903 runnable  

"GC Thread#7" os_prio=31 cpu=923.14ms elapsed=163599.12s tid=0x00007fb65e34b800 nid=0x6a03 runnable  

"GC Thread#8" os_prio=31 cpu=915.05ms elapsed=163599.12s tid=0x00007fb65ea2c000 nid=0x6b03 runnable  

"GC Thread#9" os_prio=31 cpu=885.07ms elapsed=163599.12s tid=0x00007fb65e34c000 nid=0x9403 runnable  

"GC Thread#10" os_prio=31 cpu=929.84ms elapsed=163599.12s tid=0x00007fb65e50d000 nid=0x9303 runnable  

"GC Thread#11" os_prio=31 cpu=900.90ms elapsed=163599.12s tid=0x00007fb62e05e000 nid=0x9103 runnable  

"GC Thread#12" os_prio=31 cpu=803.19ms elapsed=163492.84s tid=0x00007fb61e28a800 nid=0x1480b runnable  

"G1 Main Marker" os_prio=31 cpu=1.78ms elapsed=163599.57s tid=0x00007fb64e04a800 nid=0x3003 runnable  

"G1 Conc#0" os_prio=31 cpu=193.78ms elapsed=163599.57s tid=0x00007fb64e04b800 nid=0x5103 runnable  

"G1 Conc#1" os_prio=31 cpu=193.88ms elapsed=163598.62s tid=0x00007fb60e082000 nid=0x8e03 runnable  

"G1 Conc#2" os_prio=31 cpu=195.82ms elapsed=163598.62s tid=0x00007fb63e191000 nid=0x8c03 runnable  

"G1 Refine#0" os_prio=31 cpu=15.78ms elapsed=163599.56s tid=0x00007fb64e180000 nid=0x5003 runnable  

"G1 Refine#1" os_prio=31 cpu=7.41ms elapsed=163584.21s tid=0x00007fb61e9bf000 nid=0x13b07 runnable  

"G1 Refine#2" os_prio=31 cpu=5.63ms elapsed=163584.21s tid=0x00007fb5fe10d800 nid=0xc507 runnable  

"G1 Refine#3" os_prio=31 cpu=4.55ms elapsed=163584.21s tid=0x00007fb5be011800 nid=0x14413 runnable  

"G1 Refine#4" os_prio=31 cpu=3.09ms elapsed=163584.21s tid=0x00007fb65ebd7000 nid=0xc603 runnable  

"G1 Refine#5" os_prio=31 cpu=2.52ms elapsed=163584.21s tid=0x00007fb5de808800 nid=0x13903 runnable  

"G1 Refine#6" os_prio=31 cpu=2.37ms elapsed=163583.07s tid=0x00007fb5de816000 nid=0x13703 runnable  

"G1 Refine#7" os_prio=31 cpu=1.97ms elapsed=163583.07s tid=0x00007fb5de009000 nid=0x13503 runnable  

"G1 Refine#8" os_prio=31 cpu=1.33ms elapsed=163583.07s tid=0x00007fb5e808f800 nid=0xc803 runnable  

"G1 Refine#9" os_prio=31 cpu=0.29ms elapsed=163583.07s tid=0x00007fb59e009000 nid=0x13203 runnable  

"G1 Refine#10" os_prio=31 cpu=0.40ms elapsed=163583.07s tid=0x00007fb5de818000 nid=0xca03 runnable  

"G1 Refine#11" os_prio=31 cpu=0.03ms elapsed=163583.07s tid=0x00007fb5fe10a000 nid=0xcc03 runnable  

"G1 Young RemSet Sampling" os_prio=31 cpu=26365.54ms elapsed=163599.56s tid=0x00007fb64e181000 nid=0x3503 runnable  
"VM Periodic Task Thread" os_prio=31 cpu=35135.46ms elapsed=163599.54s tid=0x00007fb65e812000 nid=0xa003 waiting on condition  

JNI global refs: 115, weak refs: 2
alexarchambault commented 4 years ago

@gkossakowski If you have a notebook reproducing this (using public Maven / Ivy repositories), don't hesitate to post it here, that would help fixing that.

gkossakowski commented 4 years ago

This is the notebook I was working on: https://github.com/gkossakowski/covid-19/blob/36c9e8e5cfc75d3286e6f993595fe0aa854ed52c/Rainier.ipynb

alexarchambault commented 4 years ago

Are you seeing this only in a transient way? Your notebook ran fine locally for me (and it seems your last run went fine too, as the cell with a .par is done running there too).

The frequent display updates when running the .parcell worry me a bit. If the jupyter client in the browser doesn't accept messages fast enough, the queues buffering them can fill up, which should then block clients trying to publish down-the-line, so that kind of call can block. Could this be what you're seeing?

If that's the case, maybe disabling progress can help, by importing com.stripe.rainier.notebook._ like

import com.stripe.rainier.notebook.{progress => _, _}