search

yahoo / storm-yarn

Storm-yarn enables Storm clusters to be deployed into machines managed by Hadoop YARN.
Other
417 stars 161 forks source link

cannot launch application. #93

Open Babbleshack opened 4 years ago

Babbleshack commented 4 years ago

I am having trouble building and running storm-yarn.

mvn clean package -DskipTests builds a jar storm-yarn-1.0-SNAPSHOT.jar

when I execute storm-yarn launch master_default.yaml, I get a Runtime exception, caused by a missing snakeyaml dependency 

Exception in thread "main" java.lang.NoClassDefFoundError: org/yaml/snakeyaml/Yaml
        at com.yahoo.storm.yarn.Util.createConfigurationFileInFs(Util.java:133)
        at com.yahoo.storm.yarn.StormOnYarn.launchApp(StormOnYarn.java:178)
        at com.yahoo.storm.yarn.StormOnYarn.launchApplication(StormOnYarn.java:359)
        at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:67)
        at com.yahoo.storm.yarn.Client.execute(Client.java:121)
        at com.yahoo.storm.yarn.Client.main(Client.java:127)
Caused by: java.lang.ClassNotFoundException: org.yaml.snakeyaml.Yaml
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
        ... 6 more

Adding the maven assembly plugin, builds a jar including all dependency jars: 

pom.xml
<plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <executions>
                    <execution>
                            <phase>package</phase>
                            <goals>
                                    <goal>single</goal>
                            </goals>
                    </execution>
            </executions>
            <configuration>
                    <descriptorRefs>
                            <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
            </configuration>
</plugin>

however, the yarn client implementation complains about double definition of master_defaults.yaml or defaults.yaml

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Found multiple master_defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/opt/storm/stormOnYARN/storm-yarn-1.0-SNAPSHOT.jar!/master_defaults.yaml, file:/opt/storm/master_defaults.yaml]
        at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:384)
        at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:424)
        at com.yahoo.storm.yarn.Config.readStormConfig(Config.java:52)
        at com.yahoo.storm.yarn.LaunchCommand.process(LaunchCommand.java:57)
        at com.yahoo.storm.yarn.Client.execute(Client.java:121)
        at com.yahoo.storm.yarn.Client.main(Client.java:127)
Caused by: java.io.IOException: Found multiple master_defaults.yaml resources. You're probably bundling the Storm jars with your topology jar. [jar:file:/opt/storm/stormOnYARN/storm-yarn-1.0-SNAPSHOT.jar!/master_defaults.yaml, file:/opt/storm/master_defaults.yaml]
        at org.apache.storm.utils.Utils.getConfigFileInputStream(Utils.java:410)
        at org.apache.storm.utils.Utils.findAndReadConfigFile(Utils.java:363)
        ... 5 more

I have placed my storm zip in hdfs:/lib/storm

$ hdfs dfs -ls /lib/storm/1.2.2/
-rw-r--r--   3 root supergroup  168876919 2020-03-31 18:44 /lib/storm/1.2.2/storm.zip

I have also placed storm on my client machine

$ ls -l /opt/storm
total 165264
drwxr-xr-x 12 root root      4096 Mar 31 18:20  storm-1.2.2
drwxr-xr-x  8 root root      4096 Mar 31 19:59  stormOnYARN

master_defaults.yaml/master.yaml/storm.yaml

master.host: "localhost"
master.thrift.port: 55058

master.initial-num-supervisors: 1
master.container.priority: 0
master.container.size-mb: 4096
master.heartbeat.interval.millis: 1000
master.timeout.secs: 1000
yarn.report.wait.millis: 10000
nimbusui.startup.ms: 10000

storm.thrift.transport: "org.apache.storm.security.auth.SimpleTransportPlugin"
nimbus.thrift.port: 55057
nimbus.queue.size: 100000
nimbus.thrift.threads: 64
nimbus.thrift.max_buffer_size: 1048576

drpc.thrift.transport: "org.apache.storm.security.auth.SimpleTransportPlugin"
drpc.port: 55056
drpc.queue.size: 128
drpc.worker.threads: 64
drpc.max_buffer_size: 1048576

ui.port: 55055

storm.messaging.transport: "org.apache.storm.messaging.netty.Context"
storm.messaging.netty.buffer_size: 1048576
storm.messaging.netty.max_retries: 100
storm.messaging.netty.min_wait_ms: 1000
storm.messaging.netty.max_wait_ms: 5000

# Configuration parameter that allows the launching machine to specify the JAVA_JOME
# used when the application is executed on the YARN cluster.
#
#storm.yarn.java_home: "/usr/java/default"
storm.yarn.java_home: "/usr/lib/jvm/java-8-openjdk-amd64"

# Configuration parameter that allows the launching machine to specify the yarn classpath
# used when the application is executed on the YARN cluster.  To find this value, run
# "yarn classpath" on the target machines.
#
#storm.yarn.yarn_classpath: ""
storm.yarn.yarn_classpath: "/opt/hadoop/etc/hadoop:/opt/hadoop/etc/hadoop:/opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/common/lib/*:/opt/hadoop/share/hadoop/common/*:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/*:/opt/hadoop/share/hadoop/hdfs/*:/opt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/*:/opt/hadoop/share/hadoop/yarn/*:/opt/hadoop/share/hadoop/mapreduce/lib/*:/opt/hadoop/share/hadoop/mapreduce/*:/opt/hadoop/share/hadoop/yarn/*:/opt/hadoop/share/hadoop/yarn/lib/*"

TLDR

I cant seem to get the correct configuration or build.