search

databricks / reference-apps

Spark reference applications
Other
656 stars 341 forks source link

Exception in thread "main" java.lang.NoClassDefFoundError:org/apache/spark/mllib/feature/HashingTF #48

Closed manishdalmiya closed 9 years ago

manishdalmiya commented 9 years ago

I am trying to run Collect.scala from Twitter Streaming Language Classifier and keep getting below error. I think i followed all the insructions but, i am still stuck with this error. I would really appreciate if someone can help.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/mllib/feature/HashingTF at com.databricks.apps.twitter_classifier.Utils$.(Utils.scala:12) at com.databricks.apps.twitter_classifier.Utils$.(Utils.scala) at com.databricks.apps.twitter_classifier.Collect$.main(Collect.scala:26) at com.databricks.apps.twitter_classifier.Collect.main(Collect.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.mllib.feature.HashingTF at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 11 more

vidaha commented 9 years ago

Hi Manish,

For this to run, it's assumed you have Spark 1.1 installed, including having the spark mllib jar version 1.1 in your classpath.

As a quick workaround, you could comment out the "provided" statements, but please don't submit a pull request for that and leave that just in your local branch. For more details about what provided does - see this link: https://github.com/databricks/spark-knowledgebase/blob/master/troubleshooting/missing_dependencies_in_jar_files.md

-Vida