search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.46k stars 767 forks source link

HiBench.DataGen failed to run successfully #417

Open ghost opened 7 years ago

ghost commented 7 years ago

Hello, I'm trying to generate data for streaming benchmarks with genSeedDataset.sh, but it stops because of a number of java.io.FileNotFoundException file:/tmp/hadoop-aldinuc/mapred/local/1487339796623/user_agents (No such file or directory), ending with

Exception in thread "main" java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:873) at HiBench.HiveData.createUserVisitsTableDirectly(HiveData.java:609) at HiBench.HiveData.generate(HiveData.java:619) at HiBench.DataGen.run(DataGen.java:19) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at HiBench.DataGen.main(DataGen.java:45) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

I attach the part of log file containing the errors, hope that helps to understand the problem.

Thanks!

errorlog.txt

ahmedfouzan commented 6 years ago

Hi, I'm facing the same problem @clamis . Did you find any solution for it?

Thanks

yefanne commented 2 years ago

Hi, I'm facing the same problem

MeAnkitaDutta commented 2 years ago

I am also facing same problem. It is given below:

patching args= Parsing conf: /home/hadoopusr/HiBench-7.1.1/conf/hadoop.conf Parsing conf: /home/hadoopusr/HiBench-7.1.1/conf/hibench.conf Parsing conf: /home/hadoopusr/HiBench-7.1.1/conf/workloads/websearch/pagerank.conf probe sleep jar: /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar start HadoopPreparePagerank bench hdfs rm -r: /usr/local/hadoop/bin/hadoop --config /usr/local/hadoop/etc/hadoop fs -rm -r -skipTrash hdfs://ankita-OptiPlex-9020:9000/HiBench/Pagerank/Input rm: `hdfs://ankita-OptiPlex-9020:9000/HiBench/Pagerank/Input': No such file or directory Submit MapReduce Job: /usr/local/hadoop/bin/hadoop --config /usr/local/hadoop/etc/hadoop jar /home/hadoopusr/HiBench-7.1.1/autogen/target/autogen-7.1.1-jar-with-dependencies.jar HiBench.DataGen -t pagerank -b hdfs://ankita-OptiPlex-9020:9000/HiBench/Pagerank -n Input -m 8 -r 8 -p 50 -pbalance -pbalance -o text at org.apache.hadoop.util.RunJar.main(RunJar.java:236) ERROR: Hadoop job /home/hadoopusr/HiBench-7.1.1/autogen/target/autogen-7.1.1-jar-with-dependencies.jar HiBench.DataGen failed to run successfully. Hint: You can goto /home/hadoopusr/HiBench-7.1.1/report/pagerank/prepare/conf/../bench.log to check for detailed log. Opening log tail for you:

at HiBench.PagerankData.generate(PagerankData.java:311)
at HiBench.DataGen.run(DataGen.java:24)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at HiBench.DataGen.main(DataGen.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
MeAnkitaDutta commented 2 years ago

I am also facing same problem. It is given below:

patching args= Parsing conf: /home/hadoopusr/HiBench-7.1.1/conf/hadoop.conf Parsing conf: /home/hadoopusr/HiBench-7.1.1/conf/hibench.conf Parsing conf: /home/hadoopusr/HiBench-7.1.1/conf/workloads/websearch/pagerank.conf probe sleep jar: /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar start HadoopPreparePagerank bench hdfs rm -r: /usr/local/hadoop/bin/hadoop --config /usr/local/hadoop/etc/hadoop fs -rm -r -skipTrash hdfs://ankita-OptiPlex-9020:9000/HiBench/Pagerank/Input rm: `hdfs://ankita-OptiPlex-9020:9000/HiBench/Pagerank/Input': No such file or directory Submit MapReduce Job: /usr/local/hadoop/bin/hadoop --config /usr/local/hadoop/etc/hadoop jar /home/hadoopusr/HiBench-7.1.1/autogen/target/autogen-7.1.1-jar-with-dependencies.jar HiBench.DataGen -t pagerank -b hdfs://ankita-OptiPlex-9020:9000/HiBench/Pagerank -n Input -m 8 -r 8 -p 50 -pbalance -pbalance -o text at org.apache.hadoop.util.RunJar.main(RunJar.java:236) ERROR: Hadoop job /home/hadoopusr/HiBench-7.1.1/autogen/target/autogen-7.1.1-jar-with-dependencies.jar HiBench.DataGen failed to run successfully. Hint: You can goto /home/hadoopusr/HiBench-7.1.1/report/pagerank/prepare/conf/../bench.log to check for detailed log. Opening log tail for you:

at HiBench.PagerankData.generate(PagerankData.java:311)
at HiBench.DataGen.run(DataGen.java:24)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at HiBench.DataGen.main(DataGen.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

The issue is resolved now. It occurred because Hibench 7.1.1 is only supported by Hadoop 2.x , not by Hadoop 3.2.2