Hadoop multi node setup

[at15]

Currently, single node hadoop is working #5 with Yarn and can run HiBench, now having the real multi node setup is the major issue

Hadoop

• [x] properly setup host
• [x] can't view resource manager Web UI ResourceManager - http://localhost:8088/
• [x] use different config and script for single and cluster
• [x] hadoop prefix

 HADOOP_PREFIX=/path/to/hadoop
 export HADOOP_PREFIX

• [x] https://github.com/njvijay/vagrant-jilla-hadoop/blob/master/resources/hadoop/hadoop.sh
• [ ] job history server is not started by start-dfs and start-yarn I guess?

Vagrant

• [x] manage host using vagrant host manager https://github.com/devopsgroup-io/vagrant-hostmanager use static ip and fixed hosts file
• [x] use private network instead of simple port forwarding
• [x] built a base box for multiple virtual machine #7
• [x] start multiple vm in parallel instead of wait one by one can't do that
  • https://www.vagrantup.com/docs/cli/up.html#_no_parallel
  • use for loop https://github.com/njvijay/vagrant-jilla-hadoop/blob/master/Vagrantfile

Extra

• [ ] jvm option
  • Configuring Environment of Hadoop Daemons https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html
• [ ] may consider using Ansible instead of raw shell scripts
  • need to solve the grub update problem

[at15]

well, it's running, so ...