search

ooyala / spark-jobserver

REST job server for Spark. Note that this is *not* the mainline open source version. For that, go to https://github.com/spark-jobserver/spark-jobserver. This fork now serves as a semi-private repo for Ooyala.
Other
344 stars 135 forks source link

Uncaught error when initializing StreamingContext. #31

Closed bamos closed 10 years ago

bamos commented 10 years ago

Hi, I'm new to Spark and would like to use your jobserver with Spark Streaming. However, I'm running into the following error when trying to create a SparkStreamingContext from the SparkContext, causing the job server to crash.

I couldn't find anything in the repo using a StreamingContext, so is this something I'm doing wrong, or are StreamingContexts not supported?

Thanks, Brandon.

Code.

object StreamingJob extends SparkJob {
  def runJob(sc: SparkContext, config: Config): Any = {
    val batchDuration = 60
    val ssc = new StreamingContext(sc, Seconds(batchDuration))
...

Stack trace.

Line 34 is val ssc... above.

> job-server[ERROR] Uncaught error from thread [JobServer-akka.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
job-server[ERROR] java.lang.VerifyError: Bad type on operand stack
job-server[ERROR] Exception Details:
job-server[ERROR]   Location:
job-server[ERROR]     org/apache/spark/streaming/ui/StreamingTab.<init>(Lorg/apache/spark/streaming/StreamingContext;)V @10: invokespecial
job-server[ERROR]   Reason:
job-server[ERROR]     Type 'org/apache/spark/ui/SparkUI' (current frame, stack[1]) is not assignable to 'org/apache/spark/ui/WebUI'
job-server[ERROR]   Current Frame:
job-server[ERROR]     bci: @10
job-server[ERROR]     flags: { flagThisUninit }
job-server[ERROR]     locals: { uninitializedThis, 'org/apache/spark/streaming/StreamingContext' }
job-server[ERROR]     stack: { uninitializedThis, 'org/apache/spark/ui/SparkUI', 'java/lang/String' }
job-server[ERROR]   Bytecode:
job-server[ERROR]     0000000: 2a2b b600 5ab6 005f 1261 b700 642a b800
job-server[ERROR]     0000010: 682a 2bb6 005a b600 5fb5 004a 2a2a b600
job-server[ERROR]     0000020: 6ab6 006e b500 4d2a 2ab6 006a b600 70b5
job-server[ERROR]     0000030: 004f 2abb 0072 592b b700 74b5 0052 2b2a
job-server[ERROR]     0000040: b600 76b6 007a 2abb 007c 592a b700 7fb6
job-server[ERROR]     0000050: 0083 2ab6 006a 2ab6 0087 b1
job-server[ERROR]
job-server[ERROR]   at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:155)
job-server[ERROR]   at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:66)
job-server[ERROR]   at my.package.StreamingJob$.runJob(App.scala:34)
job-server[ERROR]   at spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:218)
job-server[ERROR]   at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
job-server[ERROR]   at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
job-server[ERROR]   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
job-server[ERROR]   at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
job-server[ERROR]   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
job-server[ERROR]   at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
job-server[ERROR]   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
job-server[ERROR]   at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
job-server ... finished with exit code 255
velvia commented 10 years ago

Hi Brandon,

We haven't tried integrating Spark Streaming with the Job Server yet. That said, this error really looks like something at the JVM level. Which version of Spark are you running Job Server against?

On Mon, Jun 9, 2014 at 4:04 PM, Brandon Amos notifications@github.com wrote:

Hi, I'm new to Spark and would like to use your jobserver with Spark Streaming. However, I'm running into the following error when trying to create a SparkStreamingContext from the SparkContext, causing the job server to crash.

I couldn't find anything in the repo using a StreamingContext, so is this something I'm doing wrong, or are StreamingContexts not supported?

Thanks, Brandon. Code.

object StreamingJob extends SparkJob { def runJob(sc: SparkContext, config: Config): Any = { val batchDuration = 60 val ssc = new StreamingContext(sc, Seconds(batchDuration))...

Stack trace.

Line 34 is val ssc... above.

job-server[ERROR] Uncaught error from thread [JobServer-akka.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer] job-server[ERROR] java.lang.VerifyError: Bad type on operand stack job-server[ERROR] Exception Details: job-server[ERROR] Location: job-server[ERROR] org/apache/spark/streaming/ui/StreamingTab.(Lorg/apache/spark/streaming/StreamingContext;)V @10: invokespecial job-server[ERROR] Reason: job-server[ERROR] Type 'org/apache/spark/ui/SparkUI' (current frame, stack[1]) is not assignable to 'org/apache/spark/ui/WebUI' job-server[ERROR] Current Frame: job-server[ERROR] bci: @10 job-server[ERROR] flags: { flagThisUninit } job-server[ERROR] locals: { uninitializedThis, 'org/apache/spark/streaming/StreamingContext' } job-server[ERROR] stack: { uninitializedThis, 'org/apache/spark/ui/SparkUI', 'java/lang/String' } job-server[ERROR] Bytecode: job-server[ERROR] 0000000: 2a2b b600 5ab6 005f 1261 b700 642a b800 job-server[ERROR] 0000010: 682a 2bb6 005a b600 5fb5 004a 2a2a b600 job-server[ERROR] 0000020: 6ab6 006e b500 4d2a 2ab6 006a b600 70b5 job-server[ERROR] 0000030: 004f 2abb 0072 592b b700 74b5 0052 2b2a job-server[ERROR] 0000040: b600 76b6 007a 2abb 007c 592a b700 7fb6 job-server[ERROR] 0000050: 0083 2ab6 006a 2ab6 0087 b1 job-server[ERROR] job-server[ERROR] at org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:155) job-server[ERROR] at org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:66) job-server[ERROR] at my.package.StreamingJob$.runJob(App.scala:34) job-server[ERROR] at spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:218) job-server[ERROR] at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) job-server[ERROR] at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) job-server[ERROR] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42) job-server[ERROR] at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) job-server[ERROR] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) job-server[ERROR] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) job-server[ERROR] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) job-server[ERROR] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) job-server ... finished with exit code 255

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/31.

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

bamos commented 10 years ago

I'm using Spark 1.0.0, so my errors could be coming from this incompatibility. I'll try again once #30 is implemented.

velvia commented 10 years ago

Ah, OK. Do you have the choice to try out on 0.9.1? Would be great to get the feedback.

You can also try on the #30's branch that has the change for 1.0.0.

On Mon, Jun 9, 2014 at 4:35 PM, Brandon Amos notifications@github.com wrote:

I'm using Spark 1.0.0, so my errors could be coming from this incompatibility. I'll try again once #30 https://github.com/ooyala/spark-jobserver/pull/30 is implemented.

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/31#issuecomment-45557792 .

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

bamos commented 10 years ago

Hi @velvia - I forgot to mark the streaming library as "provided" and the server crashed. I'm just using sbt-revolver, so is this behavior different when an error occurs when used in deploy mode?

 job-server[ERROR] Uncaught error from thread [JobServer-akka.actor.default-dispatcher-4] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
job-server[ERROR] java.lang.NoClassDefFoundError: org/apache/spark/streaming/StreamingContext
job-server[ERROR]   at test.StreamingJob$.runJob(App.scala:26)
job-server[ERROR]   at spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:218)
job-server[ERROR]   at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
job-server[ERROR]   at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
job-server[ERROR]   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
job-server[ERROR]   at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
job-server[ERROR]   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
job-server[ERROR]   at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
job-server[ERROR]   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
job-server[ERROR]   at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
job-server[ERROR] Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.StreamingContext
job-server[ERROR]   at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
job-server[ERROR]   at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
job-server[ERROR]   at java.security.AccessController.doPrivileged(Native Method)
job-server[ERROR]   at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
job-server[ERROR]   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
job-server[ERROR]   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
job-server[ERROR]   ... 10 more
job-server ... finished with exit code 255
bamos commented 10 years ago

After marking the streaming library as provided, I can create a StreamingContext in a simple app using the correct Scala version - thanks! Posted the files I'm using below for possible future reference.

build.sbt

import AssemblyKeys._

assemblySettings

jarName in assembly := "spark-example.jar"

name := "Simple Project"

version := "1.0"

scalaVersion := "2.10.3"

// Load "provided" libraries with `sbt run`.
run in Compile <<= Defaults.runTask(
  fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "0.9.1" % "provided",
  "org.apache.spark" %% "spark-streaming" % "0.9.1",
  "org.slf4j" % "slf4j-simple" % "1.7.7", // Logging.
  "ooyala.cnd" % "job-server" % "0.3.1" % "provided"
)

resolvers ++= Seq(
  "Akka Repository" at "http://repo.akka.io/releases/",
  "Ooyala Bintray" at "http://dl.bintray.com/ooyala/maven"
)

App.scala

package test

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.streaming.{Seconds,StreamingContext}
import org.apache.spark.streaming.StreamingContext._

import spark.jobserver.{SparkJob,SparkJobValidation,SparkJobValid}
import com.typesafe.config.Config

object StreamingJob extends SparkJob {
  private val data = Array(1, 2, 3)

  def main(args: Array[String]) {
    val sc = new SparkContext("local[4]", "StreamingJob")
    val results = runJob(sc, null)
    println("Result is " + results)
  }

  override def validate(sc: SparkContext, config: Config): SparkJobValidation = {
    SparkJobValid
  }

  override def runJob(sc: SparkContext, config: Config): Any = {
    val ssc = new StreamingContext(sc, Seconds(10))
  }
}

submit.sh

#!/bin/sh

sbt assembly &> assembly.log || exit 1
curl --data-binary @target/scala-2.10/spark-example.jar localhost:8090/jars/test
curl -d "" 'localhost:8090/jobs?appName=test&classPath=test.StreamingJob'
velvia commented 10 years ago

Nice! I'll mention that during Spark Summit.

On Mon, Jun 9, 2014 at 5:22 PM, Brandon Amos notifications@github.com wrote:

Closed #31 https://github.com/ooyala/spark-jobserver/issues/31.

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/31#event-129587047.

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

skylabreddy commented 9 years ago

I am also facing the issue in spark. That is I have created twitter app and I mapped with key in program . But I need to know the how to build the application and how to get the twitter/sec. Can any one help on this , thanks . For Twitter app I used the below URL for streaming data. https://databricks-training.s3.amazonaws.com/realtime-processing-with-spark-streaming.html#twitter-credential-setup

velvia commented 9 years ago

Are you looking for help with Spark Streaming in general, or job server? Note that the Ooyala repo is not actively developed, you want spark-jobserver/spark-jobserver.

On Thu, Jun 18, 2015 at 3:45 AM, skylabreddy notifications@github.com wrote:

I am also facing the issue in spark. That is I have created twitter app and I mapped with key in program . But I need to know the how to build the application and how to get the twitter/sec. Can any one help on this , thanks .

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/31#issuecomment-113104829 .

If you are free, you need to free somebody else. If you have some power, then your job is to empower somebody else. --- Toni Morrison

skylabreddy commented 9 years ago

Hi Velvia.

I would like to setup the one of application running on Spark. I am able to run small application .

I would like to run one of complex application . Could you have any thing please help to me to run on spark. thanks .

On Fri, Jun 19, 2015 at 10:06 AM, Evan Chan notifications@github.com wrote:

Are you looking for help with Spark Streaming in general, or job server? Note that the Ooyala repo is not actively developed, you want spark-jobserver/spark-jobserver.

On Thu, Jun 18, 2015 at 3:45 AM, skylabreddy notifications@github.com wrote:

I am also facing the issue in spark. That is I have created twitter app and I mapped with key in program . But I need to know the how to build the application and how to get the twitter/sec. Can any one help on this , thanks .

— Reply to this email directly or view it on GitHub < https://github.com/ooyala/spark-jobserver/issues/31#issuecomment-113104829

.

If you are free, you need to free somebody else. If you have some power, then your job is to empower somebody else. --- Toni Morrison

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/31#issuecomment-113364799 .

Regards.

SkylabReddyInfrastructure Specialist for Cloud CoE , IBM Cloud India , +91-968-664-4000

skylabreddy commented 9 years ago

Can any one suggest my issue here. thanks.