Open francoisroyer opened 9 years ago
This seems like a dependency issue with jarred scala classes... But help from a scala pro would be appreciated
"status": "JOB LOADING FAILED", "result": { "message": "class SampleJob has interface spark.jobserver.SparkJob as super class", "errorClass": "java.lang.IncompatibleClassChangeError", "stack": ["java.lang.ClassLoader.defineClass1(Native Method)", ...
Hi,
it seems as you're extending (instead of implementing) the spark.jobserver.SparkJob interface in your SampleJob class.
Cheers,
Chris
Am 28.04.2015 um 21:28 schrieb francoisroyer notifications@github.com:
This seems like a dependency issue with jarred scala classes... But help from a scala pro would be appreciated
"status": "JOB LOADING FAILED", "result": { "message": "class SampleJob has interface spark.jobserver.SparkJob as super class", "errorClass": "java.lang.IncompatibleClassChangeError", "stack": ["java.lang.ClassLoader.defineClass1(Native Method)", ...
— Reply to this email directly or view it on GitHub.
Indeed! Thank you. Making progress. Now just have to sort out a few more kinks - job is bombing on clojure issues now: job-server[ERROR] Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj
Good luck finding that one. My tip on this: check the uberjar for the file and also check the lein dependencies using „lein deps :tree".
Cheers,
Chris
Am 28.04.2015 um 22:23 schrieb francoisroyer notifications@github.com:
Indeed! Thank you. Making progress. Now just have to sort out a few more kinks - job is bombing on clojure issues now: job-server[ERROR] Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj
— Reply to this email directly or view it on GitHub https://github.com/gorillalabs/sparkling/issues/16#issuecomment-97197425.
Feeling the pain already - this seems to lie on the Class loader side.
(clojure/core__init.class is indeed in the uberjar deployed to the job server...)
Update from the investigation.. Since the loader is choking at clojure.lang.RT.load, and that all classes are definitely present in the jar, this may be a sign that the classloader responsible for loading clj code is not correctly set in some thread.
Hi,
did you get any further on that issue? Maybe you could share your findings to the community, so others can also use the Jobserver.
Thanks!
Cheers,
Chris
Hi I haven't solved it yet in a satisfactory way. The workaround has been to create a SparkJob from Clojure and exposing it via a Scala wrapper, e.g.
(:gen-class :name spark.jobserver.CljJob :methods [#^{:static true} [validate [org.apache.spark.SparkContext com.typesafe.config.Config] String]
)
It works, but it calls for three steps: compiling clj code, then uberjarring the scala wrapper via sbt, then submitting the jar. I'll keep you guys posted if I find a better / leiningen-only workflow.
Hi,
you might want to try https://github.com/technomancy/lein-scalac
Never tried it myself, but it sounds like an improvement to that awkward process.
Cheers, Chris
Any luck or advice on running a sparkling job on Ooyala's spark-jobserver? (https://github.com/spark-jobserver/spark-jobserver) I am gen-classing the job after extending spark.jobserver.SparkJob but may be running into class version issues ("java.lang.IncompatibleClassChangeError"). Spark job server seems to handle Spark versions 1.1.0 1.2.0 and 1.3.0.