search

ddf-project / DDF

Distributed DataFrame: Productivity = Power x Simplicity For Scientists & Engineers, on any Data Engine
http://ddf.io
Apache License 2.0
168 stars 42 forks source link

Spark Tests unsuccesful #175

Open saifellafi opened 9 years ago

saifellafi commented 9 years ago

Hi,

Pulled from master. If I -DskipTests It compiles succesfully but I cannot run examples: $ ./bin/run-example io.ddf.spark.examples.RowCount Failed to find DDF examples assembly in /var/DDF/lib or /var/DDF/examples/target You need to build DDF with mvn install before running this program

Running io.ddf.spark.SparkDDFManagerTests org.apache.spark.serializer.KryoSerializer null Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.647 sec - in io.ddf.spark.SparkDDFManagerTests

Results :

Tests in error: RepresentationHandlerSuite.Can do sql queries after Transform Rserve » Spark J... TransformationHandlerTest.testTransformNativeRserve » Spark Job aborted due to...

Tests run: 79, Failures: 0, Errors: 2, Skipped: 0

[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:46 min [INFO] Finished at: 2015-08-19T14:59:07-05:00 [INFO] Final Memory: 59M/3944M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.15:test (default-test) on project ddf_spark_2.10: There are test failures. [ERROR] [ERROR] Please refer to /var/DDF/spark/target/scala-2.10/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

saifellafi commented 9 years ago

I believe the case is the firewall blocking my rServe start test.

mvn clean install

Being this the case, why isn't maven compiling DDF with -DskipTests? [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] ddf ................................................ SUCCESS [ 1.019 s] [INFO] ddf_core ........................................... SUCCESS [ 27.010 s] [INFO] ddf_jdbc ........................................... SUCCESS [ 2.645 s] [INFO] ddf_spark .......................................... SUCCESS [ 26.850 s] [INFO] ddf_examples ....................................... SUCCESS [ 4.675 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------

$ ./bin/run-example io.ddf.spark.examples.RowCount Failed to find DDF examples assembly in /apps/cmor/DDF/lib or /apps/cmor/DDF/examples/target You need to build DDF with mvn install before running this program

Very much appreciate the assistance

Huandao0812 commented 9 years ago

@saifellafi Can you post the stack trace related to the RServe error?

saifellafi commented 9 years ago

@Huandao0812 Hi, can you assist me helping you, to do so? I am not much of a maven person.

I am not much worried about this test, but I am failing to understand why is it not compiling enough to run the examples with skip tests. Let me know if the following helps:

Running io.ddf.spark.content.RepresentationHandlerSuite rmr: DEPRECATED: Please use 'rm -r' instead. Deleted file:///tmp/hive/warehouse/airline_delayed StartRserve: first connect try failed with: Cannot connect: Connection refused StartRserve: waiting for Rserve to start ... (java.lang.UNIXProcess@7b7f52b4) StartRserve: Rserve>Error in library(Rserve) : there is no package called ‘Rserve’ StartRserve: Rserve>Execution halted StartRserve: call terminated, let us try to connect ... StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: waiting for Rserve to start ... (java.lang.UNIXProcess@45fe86d0) StartRserve: Rserve>Error in library(Rserve) : there is no package called ‘Rserve’ StartRserve: Rserve>Execution halted StartRserve: call terminated, let us try to connect ... StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: Try failed with: Cannot connect: Connection refused StartRserve: Try failed with: Cannot connect: Connection refused Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.585 sec <<< FAILURE! - in io.ddf.spark.content.RepresentationHandlerSuite Can do sql queries after Transform Rserve(io.ddf.spark.content.RepresentationHandlerSuite) Time elapsed: 5.888 sec <<< ERROR! org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 68.0 failed 1 times, most recent failure: Lost task 0.0 in stage 68.0 (TID 68, localhost): java.lang.RuntimeException: Unable to start Rserve at io.ddf.spark.etl.TransformationHandler$$anonfun$4.apply(TransformationHandler.scala:139) at io.ddf.spark.etl.TransformationHandler$$anonfun$4.apply(TransformationHandler.scala:136) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$10.next(Iterator.scala:312) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) at scala.collection.AbstractIterator.to(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) at org.apache.spark.rdd.RDD$$anonfun$33.apply(RDD.scala:1177) at org.apache.spark.rdd.RDD$$anonfun$33.apply(RDD.scala:1177) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1498) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1498) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:64) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)