There many failed workloads in Mllib test.
Such as glm-regression. The following is the error log. I use spark 1.3.0 Can anyone help to find the
reason?
16/07/23 10:50:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.spark.ml.attribute.Attribute
at mllib.perf.GLMRegressionTest.createInputData(MLAlgorithmTests.scala:119)
at mllib.perf.TestRunner$$anonfun$2.apply(TestRunner.scala:67)
at mllib.perf.TestRunner$$anonfun$2.apply(TestRunner.scala:66)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at mllib.perf.TestRunner$.main(TestRunner.scala:66)
at mllib.perf.TestRunner.main(TestRunner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.ml.attribute.Attribute
at java.net.URLClassLoader.findClass(URLClassLoader.java:600)
at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:786)
at java.lang.ClassLoader.loadClass(ClassLoader.java:764)
at java.lang.ClassLoader.loadClass(ClassLoader.java:741)
... 19 more
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
There many failed workloads in Mllib test. Such as glm-regression. The following is the error log. I use spark 1.3.0 Can anyone help to find the reason?
16/07/23 10:50:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.spark.ml.attribute.Attribute at mllib.perf.GLMRegressionTest.createInputData(MLAlgorithmTests.scala:119) at mllib.perf.TestRunner$$anonfun$2.apply(TestRunner.scala:67) at mllib.perf.TestRunner$$anonfun$2.apply(TestRunner.scala:66) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at mllib.perf.TestRunner$.main(TestRunner.scala:66) at mllib.perf.TestRunner.main(TestRunner.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56) at java.lang.reflect.Method.invoke(Method.java:620) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.ml.attribute.Attribute at java.net.URLClassLoader.findClass(URLClassLoader.java:600) at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:786) at java.lang.ClassLoader.loadClass(ClassLoader.java:764) at java.lang.ClassLoader.loadClass(ClassLoader.java:741) ... 19 more
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Java options: -Dspark.storage.memoryFraction=0.66 -Dspark.serializer=org.apache.spark.serializer.JavaSerializer -Dspark.locality.wait=60000000 -Dspark.shuffle.manager=SORT Options: glm-regression --num-trials=10 --inter-trial-wait=3 --num-partitions=64 --random-seed=5 --num-examples=500000 --num-iterations=20 --optimizer=auto --reg-type=elastic-net --elastic-net-param=0.0 --reg-param=0.01 --feature-noise=1.0 --step-size=0.0 --label-noise=0.1 --intercept=0.2 --loss=l2 --num-features=10000