search

imglib / imglib2

A generic next-generation Java library for image processing
http://imglib2.net/
Other
302 stars 91 forks source link

Cannot use LoopBuilder in Spark cluster environment #241

Closed hanslovsky closed 5 years ago

hanslovsky commented 5 years ago

Sample error message:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 64 in stage 0.0 failed 4 times, most recent failure: Lost task 64.3 in stage 0.0 (TID 200, 10.36.110.13, executor 1): java.lang.NoClassDefFoundError: net/imglib2/Sampler
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
        at java.lang.Class.getConstructor0(Class.java:3075)
        at java.lang.Class.getConstructor(Class.java:1825)
        at net.imglib2.loops.ClassCopyProvider.newInstanceForKey(ClassCopyProvider.java:130)
        at net.imglib2.loops.LoopBuilder$RunnableFactory.bindActionToSamplers(LoopBuilder.java:186)
        at net.imglib2.loops.LoopBuilder.forEachPixel(LoopBuilder.java:120)
        at net.imglib2.algorithm.labeling.affinities.Watersheds$Companion.constructAffinitiesWithCopy(Watersheds.kt:94)
        at net.imglib2.algorithm.labeling.affinities.Watersheds$Companion.constructAffinities(Watersheds.kt:80)
        at net.imglib2.algorithm.labeling.affinities.Watersheds.constructAffinities(Watersheds.kt)
        at org.janelia.saalfeldlab.label.spark.affinities.SparkRain.lambda$run$63da02bf$1(SparkRain.java:424)
        at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
        at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:939)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:939)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: net.imglib2.Sampler
        at java.lang.ClassLoader.findClass(ClassLoader.java:530)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 35 more

Might be related to #234

hanslovsky commented 5 years ago

Fixed in #234