"Open Logs" doesnt work with dfs.nameservices name

[sukh]

When clicking on "Open Logs" link on a task's information on the profile page, if I click on open logs I end up with "HTTP Status 500 - java.lang.reflect.InvocationTargetException". The root cause is

java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1

If I change the url in the address bar at this point to NamenodeHostname:8020 it works ok. i.e. the following works. i have confirmed that the node inviso is running has the correct hadoop client conf and is able to handle hdfs://nameservice1 type of urls. Any ideas how i can get inviso to be able to as well?

replacing

?fs=hdfs://nameservice1&root

with

?fs=hdfs://NamenodeHostname:8020&root

[danielcweeks]

The location is taken from the yarn-site.xml properties set in the job config. Can you check to see if the location of your yarn log aggregation directory has the correct name?

[danielcweeks]

You should be able to check the following property in the job config of any job: 'yarn.nodemanager.remote-app-log-dir'.

It will be something like: hdfs:///path/to/log/agg/dir If is missing or incorrect, it might have problems locating the file.

[sukh]

Its not that, that location referenced in the url is otherwise accessible just fine. Its just within inviso the nameservice name used in namenode HA configs doesnt seem to work correctly. I have to replace it with the currently active namenode for it to work. Seems like inviso assumes that the part after hdfs:// is a hostname. In namenode HA cases, its going to be a service name that hdfs client knows how to handle correctly but evidently inviso does not.

[danielcweeks]

This might be an issue with how inviso expects the path.

Under the covers, inviso just takes the yarn log aggregation base path and uses the log reader to construct a path. The code for how it computes the path is below (LogService.java). I don't have an HA configuration so it's a little hard to see how it's different, but you might be able to see where this breaks down in your case.

        Path logRoot = new Path(conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));

        if(root != null) {
            logRoot = new Path(root);
        }

        Path logPath = LogAggregationUtils.getRemoteNodeLogFileForApp(
                logRoot,
                ConverterUtils.toApplicationId(appId),
                owner,
                ConverterUtils.toNodeId(nodeId),
                LogAggregationUtils.getRemoteNodeLogDirSuffix(conf)
        );

        AggregatedLogFormat.LogReader reader = new AggregatedLogFormat.LogReader(conf, logPath);

[synfinatic]

So the problem is that you're creating a new empty Configuration object and it's missing the necessary options for the mapping to the actual hosts/ports. Good overview of the necessary Configuration options here:

http://www.cloudera.com/documentation/cdh/5-1-x/CDH5-High-Availability-Guide/cdh5hag_hdfs_ha_software_config.html