search

avast / hdfs-shell

HDFS Shell is a HDFS manipulation tool to work with functions integrated in Hadoop DFS
Apache License 2.0
150 stars 33 forks source link

How to run hdfs-shell into a kerberized environment ? #18

Closed jmercier-lbi closed 5 years ago

jmercier-lbi commented 5 years ago

Dear,

When trying to list file on my hadoop FS I got this error:

$ ls
ls: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]

I tried without success to provides the krb5.conf

$ JAVA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf' hdfs-shell
$ ls
ls: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]

If I add ranger-plugins-audit as dependencies that works:

diff --git a/build.gradle b/build.gradle
index c451291..a18ad6f 100644
--- a/build.gradle
+++ b/build.gradle
@@ -79,6 +79,10 @@ dependencies {

     compile "org.apache.commons:commons-lang3:3.3.2"
     compile "org.apache.hadoop:hadoop-client:2.6.0"
+    compile "org.apache.ranger:ranger-plugins-audit:1.0.0"

So, should I fix by adding the dependencies as above or they are a trick to do with the CLASSPATH when I run hdfs-shell ?

Thanks

Best regards

Vity01 commented 5 years ago

We checked HDFS-Shell running with Kerberos for CDH without any problem. That's probably related to your specific environment configuration. You can update .sh launch scripts to add any other libraries to hdfs-shell classpath, it's not necessary to update build script.