This is a script to help you quickly setup a dynamic YARN cluster on EC2. It adapts a cloud workflow to use S3 for distributed data storage and YARN for computing.
Features
To start a cluster, the script follows two steps: (1) start master machine; (2) add slaves. This two step procedure allows you to add and remove slaves on the fly.
./yarn-ec2 -k mykey -i mypem.pem launch cluster-name
./yarn-ec2 -k mykey -i mypem.pem -s nslave addslave cluster-name
./yarn-ec2 -k mykey -i mypem.pem -s nslave addspot cluster-name
--spot-price
option.Because the cluster is dynamic, all the nodes are only used as computing nodes. HDFS is only started on the master machine for temp code transfer. Normally S3 is used instead for distributed storage.
You can modify custom_master_install
and custom_all_nodes_install
in bootstrap.py
to add the packages you like to install on each machine.
In case you stopped the master and restart it on the EC2. There is no need to do the launch step again.
Instead, log into the master machine, and run startup.sh
on the home folder.
After the startup is finished, you can continue with the steps of adding slaves.
Part of yarn-ec2 is adopted from spark-ec2 script.
Most existing cluster launch script follows a start and deploy way.
These scripts requires master to be aware of the slaves and are hard to dynamically add or remove nodes. yarn-ec2 uses another way, where the master does not need to be aware of slaves beforehand.