Spark multi node setup

[at15]

Problems

• [x] JAVA_HOME not set
  • http://unix.stackexchange.com/questions/6127/java-home-not-set-in-script-when-run-using-sudo
• [x] jar not found
• [x] https://spark.apache.org/docs/latest/hadoop-provided.html
• [x] Web UI worker has wrong address
  • https://issues.apache.org/jira/browse/SPARK-8982
  • the related issue is closed https://github.com/apache/spark/pull/7345
  • http://stackoverflow.com/questions/36135742/spark-master-picking-up-wrong-ip-on-windows
• [x] change the machine name to match the hosts config could be a better idea
  • https://www.vagrantup.com/docs/vagrantfile/machine_settings.html
  • [x] got hostname -f error
  • http://askubuntu.com/questions/185070/why-do-i-get-hostname-name-or-service-not-known-error
• [x] HiBench job can't execute
  • WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that work
  • http://stackoverflow.com/questions/29469462/taskschedulerimpl-initial-job-has-not-accepted-any-resources
  • it seems the resource is too low
• [ ] the difference between driver and executor?

Depend on

• Hadoop multi node #6 , HiBench requires Hadoop cluster to generate test data

Ref

• http://spark.apache.org/docs/latest/spark-standalone.html

[at15]

vagrant@perf:/usr/lib/spark/sbin$ ./start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.master.Master-1-perf.out
failed to launch: nice -n 0 /usr/lib/spark/bin/spark-class org.apache.spark.deploy.master.Master --host master.perf.at15 --port 7077 --webui-port 8080
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more
full log in /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.master.Master-1-perf.out
slave2.perf.at15: Warning: Permanently added 'slave2.perf.at15,192.168.233.20' (ECDSA) to the list of known hosts.
master.perf.at15: Warning: Permanently added 'master.perf.at15,192.168.233.18' (ECDSA) to the list of known hosts.
slave1.perf.at15: Warning: Permanently added 'slave1.perf.at15,192.168.233.19' (ECDSA) to the list of known hosts.
master.perf.at15: starting org.apache.spark.deploy.worker.Worker, logging to /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
slave2.perf.at15: starting org.apache.spark.deploy.worker.Worker, logging to /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
slave1.perf.at15: starting org.apache.spark.deploy.worker.Worker, logging to /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
master.perf.at15: failed to launch: nice -n 0 /usr/lib/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master.perf.at15:7077
master.perf.at15:   JAVA_HOME is not set
master.perf.at15: full log in /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
slave1.perf.at15: failed to launch: nice -n 0 /usr/lib/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master.perf.at15:7077
slave2.perf.at15: failed to launch: nice -n 0 /usr/lib/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://master.perf.at15:7077
slave1.perf.at15:   JAVA_HOME is not set
slave1.perf.at15: full log in /usr/lib/spark/logs/spark-vagrant-org.apache.spark.deploy.worker.Worker-1-perf.out
slave2.perf.at15:   JAVA_HOME is not set

[at15]

Spark Command: /usr/lib/jvm/jdk1.8.0_121/bin/java -cp /usr/lib/spark/conf/:/usr/lib/spark/jars/*:/usr/lib/hadoop/etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host master.perf.at15 --port 7077 --webui-port 8080
========================================
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

[at15]

all worker has wrong ip address shown in Web-UI

worker-20170311224907-10.0.2.15-37740   10.0.2.15:37740 ALIVE   1 (0 Used)  1024.0 MB (0.0 B Used)
worker-20170311224925-10.0.2.15-35442   10.0.2.15:35442 ALIVE   1 (0 Used)  1024.0 MB (0.0 B Used)
worker-20170311224925-10.0.2.15-41952   10.0.2.15:41952 ALIVE   1 (0 Used)  1024.0 MB (0.0 B Used)