Question: Why hdfs commands fail if using hdfs://<framework name>/

[awdrius]

Hi,

I have successfully built and deployed mesos hdfs on my test cluster. The issue I encountered is that hdfs commands need the full path and port to work instead of using hdfs framework name. E.g. I left the default value of hdfs for framework name so I expected ./bin/hdfs dfs -ls hdfs://hdfs/ (as per readme) to do its thing. It works if I use the ./bin/hdfs dfs -ls hdfs://<namenode1 ip>:50071/ but that kind of defeats the purpose of HA as there needs to be external tracker of where nn1 and nn2 are, etc.

I've just started experimenting with mesos so I might be missing something. Does it need mesos-dns for framework name approach to work?

[elingg]

Does hadoop fs -ls hdfs://hdfs/ work? We mount hadoop on the /usr/bin path through use of symbolic links, but we may also want to mount the hdfs dir. It should not need mesos-dns.

[awdrius]

@elingg Ahh I think I know what's happening. I was trying to access hdsf from one of mesos master nodes (by using script in build bin directory (hdfs, hdfs.cmd, etc.)) but now I assume that I should have tested it on one of the nn, jn or dn nodes. I have shutdown the framework so I'm going to re-deploy it tomorrow and will post an update.

[elingg]

Ah, yes, the commands work from the agent nodes, not the master node. :) Common mistake. Do you feel this is clear in the docs?

[awdrius]

I can confirm that hadoop fs commands are working well on any node where hdfs nodes are deployed (be it namenode, datanode, etc.).

TBH I do not remember seeing any mention in docs about where certain commands should be executed.

At the moment I'm puzzling how to make hdfs easily accessible from the outside of mesos cluster (or on mesos master nodes at least). Also I'm thinking about chicken&egg problem: if I push hdfs mesos build into hdfs and then schedule it to run by marathon - what are the chances that one or two mesos nodes going down will make hdfs inaccessible (thus mesos hdfs build out of reach) and the whole hdfs going down...

Is there a better channel to ask questions like that? I'm a software dev so I'm thinking about writing some tooling to make my life easier (-.

[theresiasnow]

I'm getting connection errors trying just that

root@app4.lab.blue.bisnode.net:~# hadoop fs -ls hdfs://hdfs/
ls: Call From app4.lab.blue.bisnode.net/10.131.8.24 to hdfs:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

[awdrius]

@sigmundlundgren Are you executing that command on one of the nodes where either namenode or datanode are running?

[theresiasnow]

tried them all..

[awdrius]

@sigmundlundgren Have you changed mesos.hdfs.framework.name by any chance? Also, have you tried ./bin/hdfs dfs -ls hdfs://<namenode1 ip>:50071/ ?

[theresiasnow]

hi, no the framework name is left at hdfs, your suggestion works on some nodes specifying just that. Maybe it doesn't matter in reality

[elingg]

Hi @sigmundlundgren with the proper settings, the hadoop fs commands should be working well on any node where hdfs nodes are deployed (be it namenode, datanode, etc.). Were there any changes to the default settings other than specifying mesos.hdfs.zkfc.ha.zookeeper.quorum?

[yogeshnath]

Got it fixed. Looks like hdfs needed at least 5 nodes with 2 CPU and 8G RAM...

I installed hdfs on mesos HA cluster. All 3 journalnodes came up but nothing else. hdfs kills on one node and then starts on another (I think Marathon is probably trying to keep it alive)

hadoop fs -ls hdfs://hdfs/ 16/02/05 07:36:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/02/05 07:36:14 WARN hdfs.DFSUtil: Namenode for hdfs remains unresolved for ID nn1. Check your hdfs-site.xml file to ensure namenodes are configured properly. 16/02/05 07:36:14 WARN hdfs.DFSUtil: Namenode for hdfs remains unresolved for ID nn2. Check your hdfs-site.xml file to ensure namenodes are configured properly. -ls: java.net.UnknownHostException: namenode1.hdfs.mesos Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [ ...]