(LDA)Example/LDADriver/ Job aborted due to stage failure: java.lang.ArrayIndexOutOfBoundsException: -6

[ylqfp]

Example/LDADriver

Job aborted due to stage failure: Task 9 in stage 28.1 failed 4 times, most recent failure: Lost task 9.3 in stage 28.1 (TID 355, cloud1014121118.wd.nm.ss.nop.ted): java.lang.ArrayIndexOutOfBoundsException: -6 at org.apache.spark.graphx2.impl.EdgePartition.dstIds(EdgePartition.scala:114) at org.apache.spark.graphx2.impl.EdgePartition$$anon$1.next(EdgePartition.scala:341) at org.apache.spark.graphx2.impl.EdgePartition$$anon$1.next(EdgePartition.scala:333) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at org.apache.spark.graphx2.impl.EdgePartition$$anon$1.foreach(EdgePartition.scala:333) at org.apache.spark.graphx2.impl.RoutingTablePartition$.edgePartitionToMsgs(RoutingTablePartition.scala:58) at org.apache.spark.graphx2.VertexRDD$$anonfun$4$$anonfun$apply$2.apply(VertexRDD.scala:359) at org.apache.spark.graphx2.VertexRDD$$anonfun$4$$anonfun$apply$2.apply(VertexRDD.scala:359) at scala.Function$$anonfun$tupled$1.apply(Function.scala:77) at scala.Function$$anonfun$tupled$1.apply(Function.scala:76) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

[ylqfp]

My dataset is libsvm style: 0 0:3 1:2 2:10 ... 1 0:1 1:1 2:3 ... .....

Is it a bug or something wrong with my dataset?

Thanks!

[ylqfp]

Following the instructions from google, I enlarge the memory, however still failed. @witgo

[witgo]

ping @bhoppi

[ylqfp]

@bhoppi @hucheng Help!

[bhoppi]

I can't get useful info from your log. Can you dig your spark log for more detail? And please share your cmd parameters.

[ylqfp]

spark-submit --master yarn-client --class com.github.cloudml.zen.examples.ml.LDADriver \ --jars ml/target/zen-ml_2.10-0.3-SNAPSHOT.jar \ --executor-memory 6G --driver-memory 6G --num-executors 200 --executor-cores 1 \ --conf spark.driver.maxResultSize=6G \ --conf spark.driver.extraJavaOptions="-XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError" \ --conf spark.yarn.am.memory=2g \ --conf spark.yarn.am.extraJavaOptions="-XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled" \ --conf spark.storage.memoryFraction=0.1 \ --conf spark.yarn.executor.memoryOverhead=6666 \ --conf spark.sql.shuffle.partitions=2000 \ --conf spark.executor.extraJavaOptions="-XX:MaxPermSize=256m -XX:+CMSClassUnloadingEnabled -XX:MaxDirectMemorySize=2048m -Xmn100m -XX:MaxTenuringThreshold=1 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=10 -XX:+UseCompressedOops -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc.log" \ examples/target/zen-examples_2.10-0.3-SNAPSHOT.jar \ -numTopics=1000 \ -alpha=0.1 \ -beta=0.01 \ -alphaAS=0.01 \ -totalIter=50 \ -numPartitions=20 \ -useKryo=true \ -ignoredocid=true \ /user/distml/ldatest/input4 \ /user/distml/ldatest/output2

[ylqfp]

[Uploading log.txt…]()

[ylqfp]

The log.txt is a little big, so a attched it in previous post. Tell me if you cannot see the file.

[bhoppi]

Sorry I can't read the log file.

[ylqfp]

gclog.txt log.txt

[ylqfp]

Upload done... @bhoppi

[bhoppi]

@ylqfp Can you upload the container log? I can't still get the point from the master log.

[ylqfp]

Dear Bhoppi, Sorry for the late response! I use yarn logs -applicationID to get container log, however got nothing. Could you please tell me where to find the container log? Thanks!