Issue for providing an always available namenode URI to other framework like Myriad

[tangzhankun]

Problem statement:

In current design, hdfs-mesos will launch two tasks for namenode, aka namenode1 and namenode2. And mesos-DNS will generate "namenode1.hdfs.mesos" and "namenode2.hdfs.mesos" as DNS records. If we set "hdfs://namenode1.hdfs.mesos:50070/hdfs" in Myriad's configuration, it works well. But once namenode1 host lost, the active namenode will be namenode2.hdfs.mesos and the users cannot use hdfs with the old URI anymore. Is there a way to let users just specify one URI without knowing which is active namenode?

Possible Solutions:

1. If we can update the Task Name in Mesos, then we can modify the newly active namenode2 to namenode1, then mesos-DNS should update the DNS records. My concern here is this update is not supported by Mesos.
2. If mesos-DNS can support updating the DNS record, then hdfs-mesos can handle this. But will mesos-DNS like to be visible to other framework? I don't know.

Maybe I have no idea of some existing good workarounds. Hope to hear your advice.

[elingg]

The way we integrate with Spark, for example, is we use Mesos DNS and use an hdfs-site.xml file with Mesos DNS specifications, i.e. https://github.com/mesosphere/hdfs/blob/master/example-conf/mesosphere-dcos/hdfs-site.xml. This file is added as part of our Spark packaging. I assume there should be a similar way to configure Myriad, but would have to investigate further. @adam-mesos, any thoughts?

The other option is to keep track of the Active NN and provide a RSET API to retrieve it. I think we should make this part of our live or persistent state, and we would need to be sure to update it whenever it changes.

Updating the DNS record is also a cool idea. Currently, Mesos DNS doesn't allow us to do this, however. Updating the task names seems somewhat hacky to me, but it would also be a solution that could work.

[tangzhankun]

Hi, @elingg yes. the way is similar. Here is what I have done to configure Myriad ( in its Resource Manager host ) to use hdfs-mesos:

1. Replace hdfs-site.xml of Myriad's YARN resource manager with the hdfs-mesos generated hdfs-site.xml
2. Set "fs.defaultFS" to "hdfs://namenode1.hdfs.mesos:50071/hdfs" in core-site.xml of YARN

I want to keep the "hdfs://namenode1.hdfs.mesos:50071/hdfs" constant, so no need to change the configuration for the hdfs users. If we add REST API, then YARN also need to retrieve this URI when it noticed the changes, I think this may bring in extra work and dependencies for the user, right?

And is it possible to update an already running task name in Mesos? If so, I think this maybe a solution.

[adam-mesos]

1. It is not (yet) possible to update an already running task name in Mesos, although just updating the name would be a lot less involved than replacing the task binary as suggested by MESOS-1280. Could be a good phase 1.
2. Mesos-DNS gets its data directly from the mesos master's state.json, and all client calls are read-only. Allowing clients to update task names/locations in DNS would have serious security implications compared to a read-only client connection.

Another solution might be investigating if nn1 and nn2 could just always use the same task name, so that namenode.hdfs.mesos always routes to the leading NN (maybe redirecting from standby to active). Or maybe the HDFS framework can just update the active NN in ZK, and myriad's fs.defaultFS can point to "zk://zk_node:2181/hdfs/active-namenode".

[tangzhankun]

Hi, @adam-mesos Thanks for the cool ideas! Yes. Update the task name is a good way to do this. Do you know when will the phase 1 of MESOS-1280 be implemented? But at present, I'll check if nn1 and nn2 could just always use the same task name first.

[elingg]

Hi @tangzhankun, I have a simpler solution for your use case, which requires no changes:

1. Replace hdfs-site.xml of Myriad's YARN resource manager with the hdfs-mesos generated hdfs-site.xml
2. Set "fs.defaultFS" to "hdfs:///hdfs/" in core-site.xml of YARN

As a separate note, we also may want to store the active NN in ZK for overall design reasons. It may help us with different features we need.

[tangzhankun]

Hi @elingg , our case can be deployed like this. But there is a problem: If the two original name nodes failed one by one, then hdfs-mesos will start two new name nodes which will make a different hdfs-site.xml with Yarn's. How Yarn could find the hdfs-mesos again automatically?

[elingg]

The hdfs-site.xml will not be different if you use Mesos DNS, because the Mesos DNS names will resolve to the new tasks.

[tangzhankun]

hi @elingg , I don't quite understand. I checked the hdfs-site.xml generated when using Mesos-DNS. We populate the namenode infomations using hostname (not using DNS name like "namenode1.hdfs.mesos"):

dfs.ha.namenodes.hdfs nn1,nn2 dfs.namenode.rpc-address.hdfs.nn1 zhankun-ms01:50071 dfs.namenode.http-address.hdfs.nn1 zhankun-ms01:50070 dfs.namenode.rpc-address.hdfs.nn2 zhankun-ms04:50071 dfs.namenode.http-address.hdfs.nn2 zhankun-ms04:50070

Maybe my question is not clear, think Myriad use this hdfs-site.xml at first, but then the zhankun-ms01 and zhankun-ms04 failed one by one. Then the working namenodes maybe changed to zhakun-ms07 and zhankun-ms08. How could Myriad got the new hdfs-site.xml or got noticed?

I think if we can populate these with DNS names, seems the users didn't need to update the config file. Right ?

[elingg]

You need to use HDFS with Mesos DNS enabled, see the instructions here: https://github.com/mesosphere/hdfs#mesos-dns-custom-configuration.

[tangzhankun]

hi, @elingg Yes. I see that sample hdfs-site.xml and core-site.xml. Thanks. I mean hdfs-Mesos can also generated a hdfs-site.xml with "namenode1.hdfs.mesos:50071" in it. Isn't it?

[elingg]

@tangzhankun, yes, and in that case even if the namenode1 relocates, it doesn't matter because Mesos DNS will resolve it to the host where it relocated.

[tangzhankun]

yes. Do you have a plan to implement this in future? If so, I am glad to help.

[elingg]

It is implemented. Please see the instructions for running with Mesos DNS and try it out!

[tangzhankun]

Is implemented? I had a Mesos DNS enabled cluster, but didn't see a hdfs-site.xml with Mesos DNS names populated in it. Shouldn't it be implemented in ServeHdfsConfigHandler? or where is it if I missed something. :)

[elingg]

Yes, it is! In fact, we use the Mesos DNS version for DCOS! Use the example config files here, https://github.com/mesosphere/hdfs/tree/master/example-conf/mesosphere-dcos. You may need to make a few more modifications to the config files for your cluster. You need to change this property to false:

`

mesos.hdfs.native-hadoop-binaries

<value>false</value>
<description>DCOS comes with pre-distributed HDFS binaries in a single-tenant environment</description>

`

[tangzhankun]

Yes. Thanks very much. My question is that I didn't see hdfs-mesos populate this config file in "ServeHdfsConfigHandler", so such config file is a manual step? or you changed the code but only for DCOS?

[elingg]

Sure, glad to help! It will populate these config files if you place these config files under etc/hadoop/ before building and running your code.

[tangzhankun]

ah, Thanks for your patience. But I didn't quite catch up with you, @elingg. :) Could you tell me in which source code files are the "populate with Mesos DNS names" implemented in ?

[elingg]

Sure, @tangzhankun, You should set the mesos DNS names in hdfs-site.xml and mesos-site.xml. This discussion is best done offline. Could you please send me an email to follow up?

[crakjie]

I have the same problem and after reading this twice I still don't understand. The hdfs-site.xml that in the /conf directory in the build of hdfs-mesos must be place in others services like Myriad or Spark(in which directory?). You say this file is generated but where can we found it? And you ask to put it in etc/hadoop of which machine ? @tangzhankun did you found a solution to this problem?

edit: I resolve this problem like this.

• used hdfs://hdfs:50071/ when reference hfds directories
• wget http://hdfs.mesos:31542/hdfs-site.xml each time I start my app ( Spark scheduler)
• And put this completed hdfs-site.xml file into the directory specify in spark-en.sh

This way the hdfs-site file allow spark to find hdfs://hdfs without pointing to an explicit name node. And http://hdfs.mesos is the mesos-dns name for the hdfs-scheduler.

I think this tricks of getting the config from the hdfs-scheduler is useful enough to be noticed in the README.md.

[tangzhankun]

@crakjie Yes. Just like what you did, copying the framework generated hdfs-site.xml and use "hdfs://hdfs" in YARN is a working approach for integration with HDFS-Mesos. What I was asking before is how to generate a stable hdfs-site.xml with DNS in it. But now I think use such a hdfs-site.xml like this link when Mesos-DNS enabled is the right way. https://github.com/mesosphere/hdfs/tree/master/example-conf/mesosphere-dcos

[crakjie]

@tangzhankun What do you mean by stable? The files in conf are just template and not effective hdfs-site.xml, if you whant mesos-dns adresse replace the macro ${nn1Hostname} inside hdfs-site.xml by your namenode1 dns name. And each time you will call http://hdfs.mesos:31542/hdfs-site.xml you will have your stable urls

[tangzhankun]

@crakjie yes. I agree with your "stable url" approach. I mean HDFS-Mesos could populated a stable hdfs-site.xml from template with DNS names instead of host name but it don't need to do this since it's not necessary. People can just use the sample DCOS hdfs-site.xml recommended by elingg if Mesos-DNS enabled. It is a stable xml file and should work.