search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.46k stars 765 forks source link

Build error and not able to run #694

Open viswa72 opened 2 years ago

viswa72 commented 2 years ago

Hi Here is my environment. OS : Centos 7.9 Java : Openjdk version "1.8.0_302" OpenJDK Runtime Environment (build 1.8.0_302-b08) OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)

HiBench : 7.1 Spark : 2.4.8 Hadoop : 3.3.1 tried with 2.10.1 also Scala : 2.11

When i try to build it not able to compile mi . I have change the following file File name : sparkbench/ml/src/main/scala/com/intel/sparkbench/ml/CorrelationExample.scala

Removed comma at line number 14 (corrType: String = "pearson",)

Then am able to successfully compiled.

When am trying to run then i got the error.

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/hive/HiveContext at com.intel.hibench.sparkbench.sql.ScalaSparkSQLBench$.main(ScalaSparkSQLBench.scala:38) at com.intel.hibench.sparkbench.sql.ScalaSparkSQLBench.main(ScalaSparkSQLBench.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.hive.HiveContext at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

Would like to know what is the fix for this.

When i try with hadoop 2.10.1 the compilation is okay and i got the following error.

ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalStateException: Spark context stopped while waiting for backend at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:834) at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:201) at org.apache.spark.SparkContext.(SparkContext.scala:560) at com.intel.hibench.sparkbench.micro.ScalaSleep$.main(ScalaSleep.scala:31) at com.intel.hibench.sparkbench.micro.ScalaSleep.main(ScalaSleep.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

But my spark and hadoop is running.

Please let me the correct source code to use and if any steps to run.