spring-yarn examples not finding / using hadoop classpath

[quux00]

I'm trying to run the yarn examples. I tried both simple-command and batch-files on a Hortonworks HDP-2.1 multi-node (non-secured) cluster.

The job submits fine, but it fails with:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
    at org.springframework.yarn.launch.AbstractCommandLineRunner.<clinit>(AbstractCommandLineRunner.java:60)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 1 more

The only modifications to the application-context.xml and appmaster-context.xml was to edit the paths to where I copied the jars that get built with gradlew. For example, here is (part of) the simple-master application-context.xml:

    <yarn:configuration>
            fs.defaultFS=${hd.fs}
            yarn.resourcemanager.address=${hd.rm}
            fs.hdfs.impl=org.apache.hadoop.hdfs.DistributedFileSystem
    </yarn:configuration>

    <yarn:localresources>
            <yarn:hdfs path="/user/u070072/spring-yarn/app/simple-command/*.jar"/>
            <yarn:hdfs path="/user/u070072/spring-yarn/lib/*"/>
    </yarn:localresources>

    <yarn:environment>
            <yarn:classpath use-yarn-app-classpath="true"/>
    </yarn:environment>

    <util:properties id="arguments">
            <prop key="container-count">4</prop>
    </util:properties>

    <yarn:client app-name="simple-command">
            <yarn:master-runner arguments="arguments"/>
    </yarn:client>

and the appmaster-context.xml:

    <yarn:configuration>
            fs.defaultFS=${hd.fs}
            yarn.resourcemanager.address=${hd.rm}
            fs.hdfs.impl=org.apache.hadoop.hdfs.DistributedFileSystem
    </yarn:configuration>

   <yarn:localresources>
            <yarn:hdfs path="/user/u070072/spring-yarn/app/simple-command/*.jar"/>
            <yarn:hdfs path="/user/u070072/spring-yarn/lib/*"/>
    </yarn:localresources>

    <yarn:environment>
            <yarn:classpath use-yarn-app-classpath="true" delimiter=":">
                    ./*
            </yarn:classpath>
    </yarn:environment>

    <yarn:master>
            <yarn:container-allocator/>
            <yarn:container-command>
                    <![CDATA[
                    date
                    1><LOG_DIR>/Container.stdout
                    2><LOG_DIR>/Container.stderr
                    ]]>
            </yarn:container-command>
    </yarn:master>

I invoked it with:

$ ./gradlew -q run-yarn-examples-simple-command -Dhd.fs=hdfs://trvlapp0049:8020 \
-Dhd.rm=http://trvlapp0050.tsh.thomson.com:8050 -Dlocalresources.remote=hdfs://trvlapp0049:8020

The apache-commons-logging jar it wants is in /usr/lib/hadoop/lib:

u070072@TST yarn$ ls -1 /usr/lib/hadoop/lib/ | grep commons-logging
commons-logging-1.1.3.jar

and that location is in the standard hadoop classpath on the HDP platform:

$ hadoop classpath
/etc/hadoop/conf:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/.//*:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/.//*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/.//*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/lib/hadoop-mapreduce/*:/usr/lib/tez/*:/usr/lib/tez/lib/*:/etc/tez/conf

So why isn't the spring-yarn setup finding the commons-logging jar? I've run other YARN apps (not with spring-yarn) and everything works fine.

[trisberg]

Try specifying the following properties in your <yarn:configuration>:

yarn.application.classpath=/etc/hadoop/conf,/usr/lib/hadoop/*,/usr/lib/hadoop/lib/*,/usr/lib/hadoop-hdfs/*,/usr/lib/hadoop-hdfs/lib/*,/usr/lib/hadoop-yarn/*,/usr/lib/hadoop-yarn/lib/*
mapreduce.application.classpath=/usr/lib/hadoop-mapreduce/*,/usr/lib/hadoop-mapreduce/lib/*

You can also include yarn-site.xml and mapred-site.xml on your classpath. Check the clusters yarn-site.xml/mapred-site.xml for the correct paths.

AFAIK there is no reliable way for the application to detect these classpaths, so you have to provide them.

[quux00]

Thanks that fixed the immediate problem. But that uncovered another configuration problem, so I still don't have it working.

AFAIK there is no reliable way for the application to detect these classpaths, so you have to provide them.

The way to do it is start the application with yarn jar my.jar my.YarnClient arg1 argN. That puts the Hadoop classpath into the CLASSPATH of the YarnClient, which then should put that classpath into the environment hashmap that gets passed to the Yarn AppMaster, which does the same in passing that to its child containers. It is not clear how to run these examples that way.

[chang-chao]

which then should put that classpath into the environment hashmap that gets passed to the Yarn AppMaster.

I doubt on that.

What is the value of "yarn.application.classpath" in your env,does it look like something like below? and make sure the env variables used are all set.

        $HADOOP_CONF_DIR,
        $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
        $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
        $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
        $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*

[jvalkeal]

Classpath in yarn apps is one complex monster if being honest. Passing it from client would work if environment is exactly same as in the cluster which would be the case if you run the client from one of the cluster nodes. There is no universal way to find out what is the classpath so that you could run client from any location and with any env settings.

In our io guides https://spring.io/guides?filter=yarn, we pretty much package all jars in boot executable fat jar which forces classpath to be whatever we have in that executable jar. Of course this will make your 'app' a bigger but it isolates the whole classpath if you want to use something would not work with jars already in hadoops classpath. Guava, protobuf, thrift libs are usually the ones which are really old in hadoop distros and may not work with your versions.

If you want to re-use jars from hadoop distro then in reality you always need to hardcode something because distros are different.

[chang-chao]

I think I found the reason,you can see the explanation in issue 22[https://github.com/spring-projects/spring-hadoop-samples/issues/22]