search

imglib / imglib2-algorithm

Image processing algorithms for ImgLib2
http://imglib2.net/
Other
22 stars 20 forks source link

Cannot use Gauss3 in Spark cluster environment #79

Closed hanslovsky closed 5 years ago

hanslovsky commented 5 years ago

Sample error message:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 64 in stage 0.0 failed 4 times, most recent failure: Lost task 64.3 in stage 0.0 (TID 180, 10.36.110.14, executor 11): java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundErro
        at net.imglib2.algorithm.convolution.LineConvolution.execute(LineConvolution.java:151)
        at net.imglib2.algorithm.convolution.LineConvolution.forEachIntervalElementInParallel(LineConvolution.java:135)
        at net.imglib2.algorithm.convolution.LineConvolution.process(LineConvolution.java:83)
        at net.imglib2.algorithm.convolution.AbstractMultiThreadedConvolution.process(AbstractMultiThreadedConvolution.java:55)
        at net.imglib2.algorithm.convolution.Concatenation.process(Concatenation.java:80)
        at net.imglib2.algorithm.gauss3.Gauss3.gauss(Gauss3.java:207)
        at net.imglib2.algorithm.gauss3.Gauss3.gauss(Gauss3.java:131)
        at net.imglib2.algorithm.gauss3.Gauss3.gauss(Gauss3.java:92)
        at org.janelia.saalfeldlab.label.spark.affinities.SparkRain.smooth(SparkRain.java:757)
        at org.janelia.saalfeldlab.label.spark.affinities.SparkRain.access$600(SparkRain.java:79)
        at org.janelia.saalfeldlab.label.spark.affinities.SparkRain$CropAffinities.call(SparkRain.java:823)
        at org.janelia.saalfeldlab.label.spark.affinities.SparkRain$CropAffinities.call(SparkRain.java:782)
        at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
        at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
        at scala.collection.AbstractIterator.to(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:939)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:939)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: net/imglib2/algorithm/convolution/kernel/Kernel1D
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at net.imglib2.algorithm.convolution.LineConvolution.execute(LineConvolution.java:144)

Might be related to imglib/imglib2#241 and imglib/imglib2#234

hanslovsky commented 5 years ago

Fixed in imglib/imglib2#234