search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.45k stars 762 forks source link

org.apache.hadoop.hdfs.HftpFileSystem could not be instantiated #24

Closed wwshuan closed 11 years ago

wwshuan commented 11 years ago

The workload of Kmeans always failed in the prepare phase with the following errors:

SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hibench-2.2.1/common/mahout-distribution-0.7-cdh4/examples/target/mahout-examples-0.7-job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hibench-2.2.1/common/mahout-distribution-0.7-cdh4/examples/target/dependency/slf4j-jcl-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hibench-2.2.1/common/mahout-distribution-0.7-cdh4/examples/target/dependency/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.HftpFileSystem could not be instantiated: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.DelegationTokenRenewer.(Ljava/lang/Class;)V from class org.apache.hadoop.hdfs.HftpFileSystem at java.util.ServiceLoader.fail(ServiceLoader.java:224) at java.util.ServiceLoader.access$100(ServiceLoader.java:181) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377) at java.util.ServiceLoader$1.next(ServiceLoader.java:445) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2275) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2286) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351) at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:322) at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:383) at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:281) at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:422) at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:168) at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:151) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64) at org.apache.mahout.clustering.kmeans.GenKMeansDataset.main(GenKMeansDataset.java:604) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.DelegationTokenRenewer.(Ljava/lang/Class;)V from class org.apache.hadoop.hdfs.HftpFileSystem at org.apache.hadoop.hdfs.HftpFileSystem.(HftpFileSystem.java:84) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:374) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373) ... 21 more

exec "$HADOOP_EXECUTABLE" --config $HADOOP_CONF_DIR jar ${DATATOOLS} org.apache.mahout.clustering.kmeans.GenKMeansDataset -libjars $MAHOUT_HOME/examples/target/mahout-examples-0.7-job.jar ${COMPRESS_OPT} ${OPTION}

CDH 4.3 installed with Cloudera Manager 4.6 HiBench 2.2.1 RHEL6.3 64-bit

Thanks. Vincent

AllanY commented 11 years ago

Till now we only support CDH4.1 and have not yet begun the work of porting it to CDH4.3.

wwshuan commented 11 years ago

Thanks Allan. Got it.

wwshuan commented 11 years ago

Hi Allan, I'm not sure, but just double confirm with you the version of CDH4.1. Is CDH4.1.4 supported by HiBench 2.2.1?

AllanY commented 11 years ago

Vincent, basically we have tested the running of HiBench 2.2.1 on cdh 4.1.2. So currently we cannot promise HiBench can work well on higher versions of cdh.