search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.45k stars 761 forks source link

Does Hibench Suite support Hadoop version 3.x ? #722

Closed ChaseHuangxu closed 1 year ago

ChaseHuangxu commented 1 year ago

When I use Hibench 7.0 to generate micro wordcount datasets Hadoop dfs3.2.1 by Hadoop MapReduce 3.2.1 on yarn, there is an error occur:

`ERROR: Hadoop job /home/huangxu/softwares/hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar randomtextwriter failed to run successfully. Hint: You can goto /home/huangxu/softwares/HiBench-HiBench-7.0/report/wordcount/prepare/conf/../bench.log to check for detailed log. Opening log tail for you:

Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : 错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster`

I see that the current latest Hibench version is 7.1.1, which only mentions that spark2.3/2.4 is supported, and there seems to be no mention of support for hadoop 3.x

I would like to ask if Hibench still maintains updates and whether I can contribute source code on this

ChaseHuangxu commented 1 year ago

Sorry, this seems to be caused by my misconfiguration. I solved the above problem by adding yarn.application.classpath to yarn-site.xml under ${HADOOP_HOME}