search

cloudera-labs / envelope

Build configuration-driven ETL pipelines on Apache Spark
Apache License 2.0
158 stars 89 forks source link

Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging #31

Closed 383280199 closed 5 years ago

383280199 commented 5 years ago

But when I tried to run the examples/filesystem example, I found the following error.... The pom.xml file in the project has not been changed. And the spark-streaming and spark-streaming-kafka's version is 2.2.0.cloudera2。Does someone tell me why? ERROR : [root@hadoopmaster cqx_dir]# spark-submit envelope-0.5.0.jar filesystem.conf Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set Spark1 will be picked by default 18/07/14 15:38:38 INFO EnvelopeMain: Envelope application started 18/07/14 15:38:39 INFO EnvelopeMain: Configuration loaded 18/07/14 15:38:39 INFO Runner: Steps instantiated Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at com.cloudera.labs.envelope.spark.Contexts.initializeBatchJob(Contexts.java:134) at com.cloudera.labs.envelope.spark.Contexts.getSparkSession(Contexts.java:66) at com.cloudera.labs.envelope.spark.Accumulators.(Accumulators.java:50) at com.cloudera.labs.envelope.run.Runner.initializeAccumulators(Runner.java:364) at com.cloudera.labs.envelope.run.Runner.run(Runner.java:81) at com.cloudera.labs.envelope.EnvelopeMain.main(EnvelopeMain.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:750) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

jeremybeard commented 5 years ago

@383280199 It appears that it is running under Spark 1.6 instead of Spark 2.2 because of this part of your error message:

Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default

Are you running this on CDH with the required versions from the project README?

383280199 commented 5 years ago

@jeremybeard thanks,it is ok now.