at15 / hadoop-spark-perf

(Deprecated) Hadoop and Spark CPU performance Benchmark
MIT License
0 stars 0 forks source link

Spark multi node setup #10

Closed at15 closed 7 years ago

at15 commented 7 years ago

Problems

Depend on

Ref

at15 commented 7 years ago
vagrant@perf:/usr/lib/spark/sbin$ ./start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.master.Master-1-perf.out
failed to launch: nice -n 0 /usr/lib/spark/bin/spark-class org.apache.spark.deploy.master.Master --host master.perf.at15 --port 7077 --webui-port 8080
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more
full log in /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.master.Master-1-perf.out
slave2.perf.at15: Warning: Permanently added 'slave2.perf.at15,192.168.233.20' (ECDSA) to the list of known hosts.
master.perf.at15: Warning: Permanently added 'master.perf.at15,192.168.233.18' (ECDSA) to the list of known hosts.
slave1.perf.at15: Warning: Permanently added 'slave1.perf.at15,192.168.233.19' (ECDSA) to the list of known hosts.
master.perf.at15: starting org.apache.spark.deploy.worker.Worker, logging to /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
slave2.perf.at15: starting org.apache.spark.deploy.worker.Worker, logging to /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
slave1.perf.at15: starting org.apache.spark.deploy.worker.Worker, logging to /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
master.perf.at15: failed to launch: nice -n 0 /usr/lib/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master.perf.at15:7077
master.perf.at15:   JAVA_HOME is not set
master.perf.at15: full log in /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
slave1.perf.at15: failed to launch: nice -n 0 /usr/lib/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master.perf.at15:7077
slave2.perf.at15: failed to launch: nice -n 0 /usr/lib/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master.perf.at15:7077
slave1.perf.at15:   JAVA_HOME is not set
slave1.perf.at15: full log in /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
slave2.perf.at15:   JAVA_HOME is not set
at15 commented 7 years ago
Spark Command: /usr/lib/jvm/jdk1.8.0_121/bin/java -cp /usr/lib/spark/conf/:/usr/lib/spark/jars/*:/usr/lib/hadoop/etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host master.perf.at15 --port 7077 --webui-port 8080
========================================
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at15 commented 7 years ago

all worker has wrong ip address shown in Web-UI

worker-20170311224907-10.0.2.15-37740   10.0.2.15:37740 ALIVE   1 (0 Used)  1024.0 MB (0.0 B Used)
worker-20170311224925-10.0.2.15-35442   10.0.2.15:35442 ALIVE   1 (0 Used)  1024.0 MB (0.0 B Used)
worker-20170311224925-10.0.2.15-41952   10.0.2.15:41952 ALIVE   1 (0 Used)  1024.0 MB (0.0 B Used)