search

vegas-viz / Vegas

The missing MatPlotLib for Scala + Spark
MIT License
730 stars 98 forks source link

Vegas with Livy inside Zeppelin #114

Closed maziyarpanahi closed 7 years ago

maziyarpanahi commented 7 years ago

Hi,

I am using Zeppelin (0.7.2) with Livy (0.4.0-incubating - Spark 2.2). Since Livy doesn't support %dep I have to load the package by config:

livy.spark.jars.packages org.vegas-viz:vegas_2.11:0.3.11

When I do the mentioned it always breaks my RPC:

WARN RSCClient: Client RPC channel closed unexpectedly.
WARN RSCClient: Error stopping RPC.
io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@3eea3abd(uncancellable)
        at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)
        at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
        at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)
        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
        at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
        at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:218)
        at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117)
        at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28)
        at org.apache.livy.rsc.rpc.Rpc.close(Rpc.java:307)
        at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:232)
        at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:129)
        at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:123)
        at org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:108)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:956)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:608)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:586)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:748)
17/09/26 18:24:17 INFO RSCClient: Failing pending job 322a9264-ac0c-409b-961f-9b0cb98b215b due to shutdown.
17/09/26 18:24:17 INFO InteractiveSession: Failed to ping RSC driver for session 14. Killing application.
17/09/26 18:24:17 INFO InteractiveSession: Stopping InteractiveSession 14...
17/09/26 18:24:17 INFO YarnClientImpl: Killed application application_1505910137586_0055
17/09/26 18:24:17 INFO InteractiveSession: Stopped InteractiveSession 14.

When I remove the config (Vegas) it works fine. Is there anyway I can use Vegas with Livy?

Many thanks

maziyarpanahi commented 7 years ago

Just to add more details, looking at Livy logs I can see that the package is downloaded successfully:

17/09/27 17:15:10 INFO LineBufferedStream: stdout: org.vegas-viz#vegas_2.11 added as a dependency
17/09/27 17:15:10 INFO LineBufferedStream: stdout: :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
17/09/27 17:15:10 INFO LineBufferedStream: stdout:      confs: [default]
17/09/27 17:15:11 INFO LineBufferedStream: stdout:      found org.vegas-viz#vegas_2.11;0.3.10 in central
17/09/27 17:15:12 INFO LineBufferedStream: stdout:      found org.vegas-viz#vegas-macros_2.11;0.3.10 in central
17/09/27 17:15:12 INFO LineBufferedStream: stdout:      found org.scala-lang#scala-reflect;2.11.8 in central
17/09/27 17:15:12 INFO LineBufferedStream: stdout:      found org.typelevel#macro-compat_2.11;1.1.1 in central
17/09/27 17:15:12 INFO LineBufferedStream: stdout:      found com.github.julien-truffaut#monocle-core_2.11;1.1.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.scalaz#scalaz-core_2.11;7.1.1 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.scala-lang.modules#scala-parser-combinators_2.11;1.0.2 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.scala-lang.modules#scala-xml_2.11;1.0.2 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found io.circe#circe-core_2.11;0.7.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found io.circe#circe-numbers_2.11;0.7.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.typelevel#cats-core_2.11;0.9.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.typelevel#cats-macros_2.11;0.9.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found com.github.mpilquist#simulacrum_2.11;0.10.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.typelevel#machinist_2.11;0.6.1 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.typelevel#cats-kernel_2.11;0.9.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found io.circe#circe-generic_2.11;0.7.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found com.chuusai#shapeless_2.11;2.3.2 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found io.circe#circe-parser_2.11;0.7.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found io.circe#circe-jawn_2.11;0.7.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.spire-math#jawn-parser_2.11;0.10.4 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found com.github.julien-truffaut#monocle-macro_2.11;1.1.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.scalafx#scalafx_2.11;8.0.92-R10 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.webjars.bower#vega-lite;1.2.0 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.webjars.bower#vega;2.6.5 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      [2.6.5] org.webjars.bower#vega;[2.6.0,3)
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      found org.webjars.bower#d3;3.5.17 in central
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      [3.5.17] org.webjars.bower#d3;[3.5.6,4)
17/09/27 17:15:13 INFO LineBufferedStream: stdout: downloading https://repo1.maven.org/maven2/org/vegas-viz/vegas_2.11/0.3.10/vegas_2.11-0.3.10.jar ...
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      [SUCCESSFUL ] org.vegas-viz#vegas_2.11;0.3.10!vegas_2.11.jar (103ms)
17/09/27 17:15:13 INFO LineBufferedStream: stdout: downloading https://repo1.maven.org/maven2/org/vegas-viz/vegas-macros_2.11/0.3.10/vegas-macros_2.11-0.3.10.jar ...
17/09/27 17:15:13 INFO LineBufferedStream: stdout:      [SUCCESSFUL ] org.vegas-viz#vegas-macros_2.11;0.3.10!vegas-macros_2.11.jar (5ms)
17/09/27 17:15:14 INFO LineBufferedStream: stdout: :: resolution report :: resolve 3859ms :: artifacts dl 189ms
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      :: modules in use:
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      com.chuusai#shapeless_2.11;2.3.2 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      com.github.julien-truffaut#monocle-core_2.11;1.1.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      com.github.julien-truffaut#monocle-macro_2.11;1.1.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      com.github.mpilquist#simulacrum_2.11;0.10.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      io.circe#circe-core_2.11;0.7.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      io.circe#circe-generic_2.11;0.7.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      io.circe#circe-jawn_2.11;0.7.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      io.circe#circe-numbers_2.11;0.7.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      io.circe#circe-parser_2.11;0.7.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.scala-lang#scala-reflect;2.11.8 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.scala-lang.modules#scala-parser-combinators_2.11;1.0.2 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.scala-lang.modules#scala-xml_2.11;1.0.2 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.scalafx#scalafx_2.11;8.0.92-R10 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.scalaz#scalaz-core_2.11;7.1.1 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.spire-math#jawn-parser_2.11;0.10.4 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.typelevel#cats-core_2.11;0.9.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.typelevel#cats-kernel_2.11;0.9.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.typelevel#cats-macros_2.11;0.9.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.typelevel#machinist_2.11;0.6.1 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.typelevel#macro-compat_2.11;1.1.1 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.vegas-viz#vegas-macros_2.11;0.3.10 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.vegas-viz#vegas_2.11;0.3.10 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.webjars.bower#d3;3.5.17 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.webjars.bower#vega;2.6.5 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.webjars.bower#vega-lite;1.2.0 from central in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      :: evicted modules:
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      org.scala-lang#scala-reflect;2.11.6 by [org.scala-lang#scala-reflect;2.11.8] in [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      ---------------------------------------------------------------------
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      |                  |            modules            ||   artifacts   |
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      ---------------------------------------------------------------------
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      |      default     |   26  |   4   |   2   |   1   ||   25  |   2   |
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      ---------------------------------------------------------------------
17/09/27 17:15:14 INFO LineBufferedStream: stdout: :: retrieving :: org.apache.spark#spark-submit-parent
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      confs: [default]
17/09/27 17:15:14 INFO LineBufferedStream: stdout:      2 artifacts copied, 23 already retrieved (2438kB/18ms)
maziyarpanahi commented 7 years ago

Probably the number of JARS and size of some of them are not suitable for Livy. I fixed this by putting the JARS on HDFS.