DFSIOe fails to prepare on Apache 3.0.0-alpha4

[ehiggs]

I understand that HiBench currently only supports Hadoop2. But with Hadoop 3 being released soon it makes sense to have HiBench available for developers and administrators to know that there are no major regressions that would block the release or an upgrade. In fact, if, upon investigation, you see that these are regressions in compatibility, perhaps tickets should be files against the Apache Hadoop JIRA.

When I run dfsioe on a Hadoop 3 alpha4 (4b4a6524f2df3a891e9d5486ec39f7987766d84f) cluster built from Apache HEAD I get the following:

TestFDSIO.0.0.4 Enhanced Version
17/05/31 08:57:32 INFO dfsioe.TestDFSIOEnh: nrFiles = 16
17/05/31 08:57:32 INFO dfsioe.TestDFSIOEnh: fileSize (MB) = 1
17/05/31 08:57:32 INFO dfsioe.TestDFSIOEnh: bufferSize = 4096
Exception in thread "main" java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
        at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3172)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3217)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3256)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3307)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3275)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:225)
        at org.apache.hadoop.fs.dfsioe.TestDFSIOEnh.run(TestDFSIOEnh.java:578)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
        at org.apache.hadoop.fs.dfsioe.TestDFSIOEnh.main(TestDFSIOEnh.java:628)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:153)

Environment information:

$ java -version
openjdk version "1.8.0_111"
OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
$ uname -a
Linux hadoop6 4.4.0-62-generic #83~14.04.1-Ubuntu SMP Wed Jan 18 18:10:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Thanks!

[yncxcw]

Hi @ehiggs I also have same problem

Ｗei

[yncxcw]

hi,

I have compiled HiBench with Hadoop-3.0.0-alpha3.

One thing you need to change is to add the hadoop-3.0 dependency to pom.xml at top folder:

3.0.0-alpha3 During compiling, you may have some errors because these two versions have some inconsistent API(e.g., in dfsIO benchmark) Hope this would help you. Wei

[ehiggs]

One thing you need to change is to add the hadoop-3.0 dependency to pom.xml at top folder: <hadoop.mr2.version>3.0.0-alpha3</hadoop.mr2.version>

With maven, this can generally be done using -Dhadoop.mr2.version=3.0.0-alpha3 for example. Then you don't need to edit the pom.xml and can build from the command line.

[chengaofeng1]

Hello, I also have same problem,do you have solved this problem?if have, please give me some advice, THANK YOU!

[beltran]

This seems to happen with hadoop 3. I solved it removing a hadoop-hdfs-2.4.0.jar that was in the classpath.

[qiongsheng]

@beltran where is the classpath? Hibench or hadoop. Could you provide the details? Thanks!

[beltran]

@qiongsheng I had this problem while using https://github.com/hortonworks/hive-testbench. When running the benchmarks a bunch of jars are downloaded and placed in the classpath.

You may find the responsible jar by doing find ./HiBench -name hadoop-hdfs*.jar. Rename it if something comes back and try again.

[PHILO-HE]

@beltran, I also have this issue in using hive-testbench, I move hadoop-hdfs*.jar to somewhere else. It works.

[ehiggs]

Hi. Is anyone having luck here? Thanks!

[godtree]

@ehiggs Do you have the solution about this problem?

[godtree]

Hello, I also have same problem,do you have solved this problem?if have, please give me some advice, THANK YOU!

@chengaofeng1 Do you have the solution about this problem?

[godtree]

@beltran, I also have this issue in using hive-testbench, I move hadoop-hdfs*.jar to somewhere else. It works.

But I can not find the jar in the HiBench.

[PHILO-HE]

@godtree, did you try to find the jar after building HiBench? I am not sure whether what I did is applicable to fixing your problem.

[godtree]

@godtree, did you try to find the jar after building HiBench? I am not sure whether what I did is applicable to fixing your problem.

@PHILO-HE There is not the jar after building HiBench. So it can not fix the problem. Thank you all the same.