Aws deploy v2

[kotfic]

This addresses #30 by creating roles that allow for launching, terminating and provisioning clusters with spark and hdfs

[kotfic]

@aashish24 could you please review? Thanks!

[aashish24]

Sure, I didn't realize that this was ready for review. I thought you were still looking into memory issues?

How do I test this?

[kotfic]

1. clone repo/checkout branch
2. cd deploy/aws2/
3. create a file local_vars.yml in `deploy/aws2/``

prefix: aashish_NEX

# Make sure to set private key so we can access from dynamic inventory
ansible_ssh_private_key_file: [[ PATH TO YOUR AWS PEM FILE ]]

# General ec2 variables
ec2_key_name: kotfic-NEX
ec2_instance_profile_name: NEX
ec2_region: us-west-2

# Master related variables
ec2_master_ami: ami-e0f7e081
ec2_master_instance_type: c4.2xlarge
ec2_master_volumes:
  - device_name: /dev/sda1
    volume_size: 1000 

# Make sure we're consistent with gobig ec2-pod role
ec2_master_additional_tags:
  ec2_pod: "{{ prefix }}"
  ec2_pod_instance_name: master

# Node related variables
ec2_node_ami: ami-e0f7e081
ec2_node_instance_type: c4.2xlarge
ec2_node_count: 2
ec2_node_volumes:
  - device_name: /dev/sda1
    volume_size: 1000

# Make sure we're consistent with gobig ec2-pod role
ec2_node_additional_tags:
  ec2_pod: "{{ prefix }}"
  ec2_pod_instance_name: nodes

1. export your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your terminal. Run `export CLUSTER_ID=aashish_NEX``
2. run ansible-playbook -i "localhost," -e @local_vars.yml launch.yml"
3. Once that is complete run ansible-playbook -i cumulus.py -e @local_vars.yml provision.yml
4. SSH into remote machine, verify spark/hadoop are installed in /opt
5. exit machine, run ansible-playbook -i cumulus.py -e @local_vars.yml terminate.yml

[kotfic]

I can open a separate PR for resolving memory issues - those will be system dependent anyways

[kotfic]

@aashish24 Please review.

[aashish24]

Step 2 ran fine.

Step 3, I ran into this issue. I believe it could be because the instances launched are not reporting that they are using my key?

[chaudhary@einstein aws2 (aws_deploy_v2)]$ ansible-playbook -i cumulus.py -e local_vars.yml provision.yml [WARNING]: provided hosts list is empty, only localhost is available

PLAY [Provision the master node] *** skipping: no hosts matched

PLAY [Provision the regular nodes] ***** skipping: no hosts matched

PLAY [Start services and daemons on master node] *** skipping: no hosts matched

PLAY RECAP *****

[chaudhary@einstein aws2 (aws_deploy_v2)]$

[kotfic]

two things to check:

which of the following are you running: ansible-playbook -i cumulus.py -e local_vars.yml provision.yml ansible-playbook -i cumulus.py -e @local_vars.yml provision.yml

the -e option to ansible playbook must have an arobase (@) if pulling extra variable from a file.

If you are running with the arobase, could you show me the output of running ./cumulus.py --list?

thanks!

On Fri, Apr 22, 2016 at 6:26 PM, Aashish Chaudhary <notifications@github.com

wrote:

Step 2 ran fine.

Step 3, I ran into this issue. I believe it could be because the instances launched are not reporting that they are using my key?

[chaudhary@einstein aws2 (aws_deploy_v2)]$ ansible-playbook -i cumulus.py -e local_vars.yml provision.yml [WARNING]: provided hosts list is empty, only localhost is available

PLAY [Provision the master node]

---

skipping: no hosts matched

PLAY [Provision the regular nodes]

---

skipping: no hosts matched

PLAY [Start services and daemons on master node]

---

skipping: no hosts matched

PLAY RECAP

---

[chaudhary@einstein aws2 (aws_deploy_v2)]$

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/OpenGeoscience/nex/pull/34#issuecomment-213610514

Christopher Kotfila R&D Engineer Kitware Inc. http://www.kitware.com/company/team/kotfila.html

[aashish24]

@kotfic I am running it again to send you more infomation.

[aashish24]

This time, I see this error:

[chaudhary@einstein aws2 (aws_deploy_v2)]$ ansible-playbook -i "localhost," -e @local_vars.yml launch.yml

PLAY ***************************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [Launch master instance] **************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: __init__() got an unexpected keyword argument 'encrypted'
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "parsed": false}

NO MORE HOSTS LEFT *************************************************************
    to retry, use: --limit @launch.retry

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1  

[aashish24]

@kotfic doing it from start /scratch just to make sure that I didn't have anything bad locally.

[kotfic]

What version of ansible are you using?

On Wed, Apr 27, 2016 at 12:53 PM, Aashish Chaudhary < notifications@github.com> wrote:

@kotfic https://github.com/kotfic doing it from start /scratch just to make sure that I didn't have anything bad locally.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/OpenGeoscience/nex/pull/34#issuecomment-215146507

Christopher Kotfila R&D Engineer Kitware Inc. http://www.kitware.com/company/team/kotfila.html

[aashish24]

[chaudhary@einstein aws2 (aws_deploy_v2)]$ ansible --version ansible 2.0.1.0 config file = /home/chaudhary/tools/nex/src.git/deploy/aws2/ansible.cfg configured module search path = Default w/o overrides

[aashish24]

@kotfic same error as last time for me in clean state as well.

[kotfic]

please post the contents of your local_vars.yml

make sure to remove any sensitive information.

On Wed, Apr 27, 2016 at 1:00 PM, Aashish Chaudhary <notifications@github.com

wrote:

@kotfic https://github.com/kotfic same error as last time for me in clean state as well.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/OpenGeoscience/nex/pull/34#issuecomment-215148398

Christopher Kotfila R&D Engineer Kitware Inc. http://www.kitware.com/company/team/kotfila.html

[aashish24]

@kotfic sent over the email.

[aashish24]

After I updated boto version as suggested by @kotfic everyhing worked like a charm :clap: :2 LGTM

[aashish24]

+2

[aashish24]

@kotfic I am assuming you saw my +2. Looks good on my testing so far.

[kotfic]

@aashish24 nope missed it, merging!