Hydrospheredata / mist

Serverless proxy for Spark cluster
http://hydrosphere.io/mist/
Apache License 2.0
326 stars 68 forks source link

Unable to Run Custom Spark Scala Job #375

Closed SushantVarshney closed 6 years ago

SushantVarshney commented 6 years ago

After creating jar with mist dependencies,I am registering the jar in router.conf like

custom-example = { path = "/home/hadoop/testMist.jar" className = "com.yash.testMist.demoApp$" namespace = "foo" } but I am not getting the same entry on the UI of mist in the Jobs Section and it's throwing an exception

java.lang.IllegalStateException: Can not instantiate job for action serve at io.hydrosphere.mist.core.jvmjob.JobsLoader$$anonfun$loadJobInstance$1.apply(JobsLoader.scala:24) at io.hydrosphere.mist.core.jvmjob.JobsLoader$$anonfun$loadJobInstance$1.apply(JobsLoader.scala:15) at scala.util.Success.flatMap(Try.scala:231) at io.hydrosphere.mist.core.jvmjob.JobsLoader.loadJobInstance(JobsLoader.scala:15) at io.hydrosphere.mist.job.JvmJobInfoExtractor.io$hydrosphere$mist$job$JvmJobInfoExtractor$$extractInstance(JobInfoExtractor.scala:45) at io.hydrosphere.mist.job.JvmJobInfoExtractor$$anonfun$extractInfo$1.apply(JobInfoExtractor.scala:27) at io.hydrosphere.mist.job.JvmJobInfoExtractor$$anonfun$extractInfo$1.apply(JobInfoExtractor.scala:27) at scala.util.Try.orElse(Try.scala:84)

Please look into this matter.

dos65 commented 6 years ago

What version of mist are you using?

SushantVarshney commented 6 years ago

The above mentioned issue was resolved.But now I want to know how can I specify deploy modes(cluster or client) while running spark jobs in Yarn Mode in default.conf

dos65 commented 6 years ago

You need set it in context configuration - example

We also have a gitter room - feel free to ask help there