search

rhinempi / sparkhit

sparkhit - analyzing large scale genomic data on the cloud
https://rhinempi.github.io/sparkhit/
Apache License 2.0
6 stars 2 forks source link

java.lang.AbstractMethodError #1

Open rhinempi opened 7 years ago

rhinempi commented 7 years ago

Error log reported by Chu Wang:

17/02/08 18:44:20 ERROR Utils: Aborting task java.lang.AbstractMethodError: uni.bielefeld.cmg.sparkhit.pipeline.SparkPipe$2SparkBatchAlign.call(Ljava/lang/Object;)Ljava/util/Iterator; at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply$mcV$sp(PairRDDFunctions.scala:1203) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply(PairRDDFunctions.scala:1203) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply(PairRDDFunctions.scala:1203) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1325) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1211) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1190) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) at org.apache.spark.scheduler.Task.run(Task.scala:85) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 17/02/08 18:44:20 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)

rhinempi commented 7 years ago

Spark 2.0.0 version API changed return type of call function to "iterator".

Fixed

tom-dyar commented 6 years ago

Hi! I ran into this while just getting set up -- is there a workaround??

rhinempi commented 6 years ago

Hi Tom, which Spark version are you using? There is a major change on the interface of the Spark 2.0.0 version. If you are using Sparkhit on the Spark 2.0.0+, choose the Sparkhit 1.0 version. If you have set up a Spark cluster with 2.0.0- version (say 1.6.0), you can still use the Sparkhit 0.8 version by changing the sparkhit executable shell : name="sparkhit" version="1.0" # change to 0.8 if you are using Spark version under 2.0.0 spark_version="2.0.0" # only for auto downloading Spark package Let me know if you have further questions.

tom-dyar commented 6 years ago

Liren,

That fixed it , thanks!

I was also having trouble running the example in which it would complain about the input and reference files not existng so I had to copyFromLocal to to hdfs (running on a cluster backed by hdfs not standalone). Seems to be some confusion between local and hdfs files?

I will look at the code if I get a chance, it might be helpful to check for the input and reference files as well as non-empty result directory at the beginning of a run since that was disappointing to get through the run only to have it fail at the end when it couldn’t write to the result directory...

Tom

From: Liren Huang notifications@github.com Sent: Saturday, February 24, 2018 7:23:04 AM To: rhinempi/sparkhit Cc: Thomas Dyar (EXTERNAL); Comment Subject: Re: [rhinempi/sparkhit] java.lang.AbstractMethodError (#1)

Hi Tom, which Spark version are you using? There is a major change on the interface of the Spark 2.0.0 version. If you are using Sparkhit on the Spark 2.0.0+, choose the Sparkhit 1.0 version. If you have set up a Spark cluster with 2.0.0- version (say 1.6.0), you can still use the Sparkhit 0.8 version by changing the sparkhit executable shell : name="sparkhit" version="1.0" # change to 0.8 if you are using Spark version under 2.0.0 spark_version="2.0.0" # only for auto downloading Spark package Let me know if you have further questions.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rhinempi_sparkhit_issues_1-23issuecomment-2D368224827&d=DwMFaQ&c=n7UHtw8cUfEZZQ61ciL2BA&r=Wpxr3yZIgJUDc4CsPhUpuiAtTKDn_lya4DWla3Q21iI&m=oOTbRAUg9ZSSE-2ZdtH2NOM8aT7_1vlWxv-JOuiGhHw&s=bWZbVFSJ6XVFE3zjI28BYaT0wAZZdd9InpAiYVfChEA&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIfLuRHIDBwsbXiYRGTYuARhVPhdRKNcks5tX-5F8ogaJpZM4L775e&d=DwMFaQ&c=n7UHtw8cUfEZZQ61ciL2BA&r=Wpxr3yZIgJUDc4CsPhUpuiAtTKDn_lya4DWla3Q21iI&m=oOTbRAUg9ZSSE-2ZdtH2NOM8aT7_1vlWxv-JOuiGhHw&s=KySptUO-y4HAmQ3PDNud_JwMagOf7qG8k2ljpxvj8I8&e=.

rhinempi commented 6 years ago

Hi Tom,

Thank you for your advise and comments. I will update the functions in the next release.

Liren