search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.45k stars 761 forks source link

Many workloads do not work in CDH6.0.0( Hadoop3 ) #614

Open godtree opened 4 years ago

godtree commented 4 years ago

I find that the HiBench is supported the CDH5. I test the HiBench in CDH6.0.0 . But CDH 6 is based on Apache Hadoop 3. And many workloads don't work. Such as enhanced DFSIO, Bayesian Classification, K-means clustering, Scan, Join, Aggregate, Nutch indexing, and so on.

The Errors show like this: Exception in thread "main" java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator

When will support CDH6？When will support Hadoop 3？

gcz2022 commented 4 years ago

We may only support Hadoop 3.x after https://github.com/Intel-bigdata/HiBench/pull/586 is merged. Also, there may be a few changes(or none) you may need to do on your own to run on Spark 3.0/Spark 3.1(as that one is for Hadoop 3.2).

As for CDH 6.0, similar answer, we currently don't have plans to officially support it. But I suppose the code changes needed should not be too much.

zhizhunbao commented 4 years ago

i have the same issue

fuzhenhe commented 4 years ago

Hi, Sir

I have a question that about Hadoop version-hadoop-3.3, because I noticed that the Hadoop ARM version is already supported on the Apache website. Do you have plan working on hadoop-3.3.0-aarch64, many thanks! zfu from ampere

jasonwzs commented 4 years ago

Hi @gczsjdy , does the latest hibench release or trunk support any Hadoop 3.x version? I see the README says it support following: Hadoop: Apache Hadoop 3.0.x, Apache Hadoop 3.1.x, Apache Hadoop 3.2.x, Apache Hadoop 2.x, CDH5, HDP However #586 is not merged yet. Does it mean it still has some problem support Hadoop 3.x? Thanks!

gcz2022 commented 4 years ago

@jasonwzs , Hadoop 3.x(until 3.3.0, exclusive) is supported, please try out. #586 was the old story. : )

gcz2022 commented 4 years ago

@fuzhenhe , I am not sure if the ARM part is the problem, I suppose JVM will take care of the ARM part? Hadoop 3.0 is the version we haven't officially support. If you have tried that out yourself, it's welcome if you left some feedback.