Closed jmercier-lbi closed 5 years ago
Do you use same Hadoop client version + same configuration (HADOOP home dir) for both hdfs-shell (2.6 by default) and the hdfs dfs client? I am not able to reproduce it.
Thanks @Vity01 for your quick answer,
I made some test and now able to provides more information.
So If I build the tools with these 2 new lines after line 184
conf.addResource(new Path("file:///etc/hadoop/conf/core-site.xml"));
conf.addResource(new Path("file:///etc/hadoop/conf/hdfs-site.xml"));
I have the real error:
Command failed java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.netapp.fs.nfs.NFSv3FileSystem not found
java.lang.ClassNotFoundException: Class org.apache.hadoop.netapp.fs.nfs.NFSv3FileSystem not found
Indeed we use a netApp technology to communicate with nfs through hdfs. Thus we have a lib named: hadoop-hdfs-nfs-3.1.1.3.1.0.0-78.jar
So should I rebuild hdfs-shell in order to use this library ? by inserting the below gradle code int build.gradle ?
configurations{
//declare custom config if necessary, otherwise just use compile
myLibs
}
dependencies {
//add lib/*.jar files to myLibs
myLibs fileTree(dir: 'lib', include: '*.jar')
compile {
//set compile configuration to extend from myLibs
extendsFrom myLibs
}
}
// task to copy libs to output/lib dir
task copyToLib(type: Copy) {
into "$buildDir/output/lib"
from configurations.myLibs
}
jar {
//include contents of output dir
from "$buildDir/output"
manifest {
//...
}
}
//set build task to depend on copyToLib
build.dependsOn(copyToLib)
Thanks
Have a nice day
Hmms, I would try to rebuild that with a hadoop client 3.1.1 which is obviously version you are using on your system.
In the build.gradle just replace this line:
compile "org.apache.hadoop:hadoop-client:2.6.0"
with compile "org.apache.hadoop:hadoop-client:3.1.1"
Rebuild it with gradlew build
.
Make sure your HADOOP_CONF_DIR
directory is set on your system (via echo $HADOOP_CONF_DIR
default value is /etc/hadoop/conf
.
Then run hdfs-shell.
Can I close the issue?
I am working on this issue:
1/ Using HADOOP_CONF_DIR do not help , I continue to investigate...
2/ about nfs
I solved it by compiling with jar provided by NetApp
diff --git a/build.gradle b/build.gradle
index c451291..c737a98 100644
--- a/build.gradle
+++ b/build.gradle
@@ -80,6 +80,8 @@ dependencies {
compile "org.apache.commons:commons-lang3:3.3.2"
compile "org.apache.hadoop:hadoop-client:2.6.0"
+ compile fileTree(include: ['*.jar'], dir: 'libs')
+
testCompile 'junit:junit:4.12'
testCompile("org.springframework.boot:spring-boot-starter-test:1.4.2.RELEASE")
}
$ ls libs
hadoop-nfs-2.7.1.jar hadoop-nfs-connector-3.0.1.jar
3/ I am looking how to provides to hdfs-shell a Kerberos token
$ hdfs-shell
foo@domain $ ls
ls: SIMPLE authentification is not enabled. Available:[TOKEN, KERBEROS]
Thanks Have a nice day
Dear,
The command below fails with nfs while the native tool
hdfs dfs
workswhere xxx.yy.zzz.aa:pppp is an IP