search

genkio / blog

Stay hungry stay foolish
https://slashbit.github.io/blog/
0 stars 1 forks source link

AWS developer getting started #142

Open genkio opened 7 years ago

genkio commented 7 years ago

study notes taken from the pluralsight AWS Developer: Getting Started course.

Application Mapping to AWS Services

image

Sounding the Alarm with IAM and Cloudwatch

Install AWS command line interface

$ pip install awscli
$ aws --version

Generate AWS access key

Create an SNS (Simple Notification Service) Topic with Email subscription

for numbered notes reading: page elements like links, tab names are capitalized, actions performed on the same page will be put in the same numbered note.

  1. Switch to US East (N. Virginia) region
  2. Services - SNS
  3. Create Topic and name it 'admin_email'
  4. Create Subscription
  5. choose email as Protocol and fill in the email as Endpoint
  6. confirm email

Enable billing alert

  1. (topic right header dropdown) My Billing Dashboard
  2. Preferences
  3. check the box Receive Billing Alerts (this checkbox is now default to checked)

Setup a CloudWatch billing alarms

(tbc) billing alerts are only enabled in the north virginia region

  1. Switch to US East (N. Virginia) region
  2. Services - CloudWatch
  3. Create Alarm
  4. select Receive Billing Alerts
  5. select USD as currency and 12h to 0
  6. set EstimatedCharges to $5
  7. select the previously created SNS topic in the Send Notification To dropdown, and select State is ALARM in the Whenever this alarm dropdown
  8. Create Alarm

Setup MFA on your root AWS account

Create a new user

  1. Services - IAM
  2. Users
  3. Add User
  4. enter user name, check Generate an Access Key for Each User checkbox
  5. Create
  6. download user credential
  7. (now you should delete the root account access key to make your account more secure)
  8. go to Security Credentials tab
  9. Manage Password
  10. Assign a custom password
  11. go to IAM dashboard home screen to get the IAM user sign-in link

Create a user group

  1. Services - IAM
  2. Groups
  3. Create New Group
  4. name it 'admin' and attach the AdministratorAccess policy
  5. Create Group
  6. Add User to Group

Configure AWS CLI

$ aws configure

AWS Access Key ID []: 
AWS Secret Access Key []: 
Default region name [ap-northeast-1]: 
Default output format [None]: json

Test AWS configuration

$ aws ec2 describe-instances

{
    "Reservations": []
}

Application architecture diagram

image

Getting Inside the Virtual Machine with EC2 and VPC

Create a VPC (Virtual Private Cloud) with 2 subnets

  1. Services - VPC
  2. Start VPC Wizard
  3. select the default VPC with a Single Public Subnet
  4. name your VPC pizza-vpc, select any (us-west-2a) Availability Zone, and name Subnet as pizza-subnet-a
  5. Create VPC

Enable VPC for accessing Internet

  1. VPC Dashboard
  2. select your created VPC
  3. click the Route Table link in the Summary tab
  4. select the Routes tab
  5. Edit
  6. add another route with Destination 0.0.0.0/0 and Target igw-xxx (the pre-created Internet gateway)
  7. Save

Create public subnet for scaling

  1. VPC Dashboard
  2. Subnets
  3. Create Subnet
  4. name it pizza-subnet-b, select our pizza in the VPC dropdown, select a different Availability Zone than the other subnet, give another CIDR block 10.0.1.0/24 to avoid IP conflict with the other subnet
  5. Yes Create

Create an EC2 instance

  1. EC2 Dashboard
  2. Launch Instance
  3. select Amazon Linux AMI
  4. select the previously created VPC in the Network dropdown, select either of the subnet
  5. select Disable in Auto-assign Public IP dropdown, leave rest of settings as default then Next
  6. name it pizza-og
  7. name the Security Group pizza-ec2-sg, leave the default access type (SSH) as it is for now, add a new Custom TCP Rule with Port Range 3000 and Source anywhere
  8. Launch
  9. Create a new Key Pair with name pizza-keys and download the pem file
  10. Launch

Create an Elastic IP and associate EC2 instance

  1. EC2 Dashboard
  2. Elastic IPs
  3. Allocate New Address
  4. Associate Address, select our pizza-og instance
  5. Associate

SSH into EC2 instance and install node

$ chmod 400 pizza-keys.pem
$ ssh -i pizza-keys.pem ec2-user@<your_ec2_elastic_ip>
$ sudo yum update
$ curl --location https://rpm.nodesource.com/setup_6.x | sudo bash -
$ sudo yum install -y nodejs
$ exit

Transfer application code onto EC2

# remove local dependencies with `rm -rf node_modules/`
$ scp -r -i ~/Downloads/pizza-keys.pem ./pizza-luvrs ec2-user@<your_ec2_elastic_ip>:/home/ec2-user/pizza-luvrs

Launch application on EC2

$ ssh -i pizza-keys.pem ec2-user@<your_ec2_elastic_ip>
$ cd pizza-luvrs && npm i
$ npm start

# now your application is available at <your_ec2_elastic_ip>:3000

Create an AMI (Amazon Machine Image)

  1. EC2 Dashboard
  2. Instances
  3. select instance you'd like to clone
  4. Actions
  5. Image
  6. Create Image
  7. give your image a name, say 'pizza-image'
  8. Create Image

Create a load balancer

  1. EC2 Dashboard
  2. Load Balancers
  3. Create Load Balancer
  4. Classic Load Balancer
  5. name it pizza-loader, select our VPC, then change the Instance Port to 3000 (the port our app is running on), last select both of our subsets
  6. Create a New Security Group, name it pizza-lb-sg, select Anywhere in the Source dropdown
  7. Configure Health Check, change the Ping Path to / instead of /index.html
  8. Leave the Add EC2 Instances as it is, next
  9. Review and Create without Add Tags

Enable instance stickiness on load balancer

  1. EC2 Dashboard
  2. select the load balancer just created, Edit Stickness
  3. Enable load balancer generated cookie stickiness, set Expiration Period to a whole day 86400

Create auto-scaling group to use with load balancer

Create Launch Configuration

  1. EC2 Dashboard
  2. Auto Scaling Groups
  3. Create Auto Scaling Group
  4. Create Launch Configuration
  5. select My AMIs, select our pizza-image AMI
  6. name it pizza-launcher, in the Advanced Details to enable our node app could be ran automatically with the script below (put in the User Data field)
  7. make no change to the storage options
  8. Select An Existing Security Group, select pizza-ec2-sg, then click Review
  9. select pizza-keys as Key Pair

  10. Create Launch Configration

    !/bin/bash

    echo "starting pizza-luvrs" cd /home/ec2-user/pizza-luvrs npm start

Create Auto Scaling Group

  1. name it pizza-scaler, set Group Size to 2 instances, select pizza-vpc as Network, then add both subnets, don't mind the 'No public IP addresses will be assigned' warning, as only of our load balancer shall be opened to the public, check Receive traffic from one or more load balancers in the Advanced Details section, select pizza-loader as Classic Load Balancers, then click Next
  2. Keep this group as its initial size, Next
  3. Skip Add Notification step for now, Review
  4. Create auto-scaling group

Secure our EC2 instance to be accessed only by load balancer

  1. Security Groups
  2. Edit the Inbound access of pizza-ec2-sg
  3. update the Custom TCP Rule with Source set to sg-xxx (pizza-lb-sg)

Configure scaling rules for scaling group

  1. EC2 Dashboard
  2. Auto Scaling Groups
  3. select pizza-scaler
  4. select Scaling Policies, and Add Policy
  5. name it scale up, Create New Alarm with Average of Network Out is >= 5000000 Bytes (for testing purpose only)
  6. set Add 1 Instance as the Take the Action option
  7. add one more policy to scaling back down, with configuration scale down when Network Out is <= 5000000 Bytes, and Remove 1 Instance as its action
  8. lastly set auto scaling group max to 4 instances

Use apache benchmark for pressure testing

# 100 requests to our load balancer at max concurrent 5 requests at a time
$ ab -n 100 -c 5 http://<url_to_load_balancer>/

Hosting All the Things with S3

Create an S3 bucket

Grand access permission to everyone

  1. go to Permissions tab of your S3 bucket
  2. click Bucket Policy link
  3. generate AWS policy with configuration, S3 Bucket Policy for type, for Principal, GetObject for Actions, arn:aws:s3:::/ for ARN. Then Add Statement, Generate Policy, copy the generated policy json, paste it to the Bucket Policy textarea, Save and done

Copy files to S3

# go to our project directory
$ aws s3 cp ./assets/js/ s3://<s3_bucket_name>/js/ --recursive --exclude ".DS_Store"

Implement the storage to S3 code logic

const AWS = require('aws-sdk');

const s3 = new AWS.S3();

module.exports.save = (name, data, callback) => {
  let params = {
    Bucket: '<s3_bucket_name>',
    Key: `pizzas/${name}.png`,
    Body: new Buffer(data, 'base64'),
    ContentEncoding: 'base64',
    ContentType: 'image/png'
  };

  s3.putObject(params, (err, data) => {
    callback(err, `<s3_bucket_root_path>/${params.Key}`)
  });
};

Work with CORS in S3

  1. go to Permissions tab of your S3 bucket
  2. click CORS configuration link
  3. Save the already filled in sample policy

Access S3 with EC2

Create a new EC2 instance

  1. EC2 Dashboard, Instances
  2. select pizza-og instance, select Create Image in the Actions dropdown
  3. name it pizza-plus-s3 then Create Image

Create a new IAM role

  1. IAM Dashboard, Role (roles are used to attach policies to)
  2. Create New Role with name pizza-ec2-role
  3. select Amazon EC2 as Role Type
  4. Select AmazonS3FullAccess as Policy to attach
  5. Create Role

Create a new launch configuration

  1. EC2 Dashboard, Launch Configuration
  2. Create Launch Configuration
  3. select My AMIs - pizza-plus-s3, name it pizza-launcher-2, select pizza-ec2-role for the IAM role dropdown, give the same start script as above for User Data, lastly assign public IP to this instance as it needs to go to public Internet to access S3 by check the Assign a Public IP Address to Every Instance option
  4. select pizza-ec2-sg for its Security Group
  5. Create Launch Configuration

Replace the old instance with the new one in the Auto Scaling Group

  1. EC2 Dashboard, Auto Scaling Group
  2. select pizza-scaler, Edit
  3. change pizza-launcher to pizza-launcher-2 in the Launch Configuration dropdown, Save
  4. Terminate all other instances that aren't pizza-og (ones could be created by auto scaling)

DynamoDB and RDS

Create (Postgres) DB instance

Create a dev db with these Advanced Settings (rest of the settings are all pretty self explanatory)

Connect to a Postgres database with Postico

  1. RDS Dashboard, Instances
  2. click the Security Groups link in the Configuration Details tab
  3. edit Inbound Source setting to Anywhere
  4. connect Postgres database with either Postico or pgAdmin using the Endpoint address which can be found in the Configuration Details tab
  5. create the db table for our app

Interact with RDS in code with using sequelize

Create DynamoDB tables

  1. Services, DynamoDB, Create Tables
  2. name your table toppings, with type String id as its Partition Key, then Create
  3. create the second table named users, with type String username as key

Connect to DynamoDB with code

Access RDS and DynamoDB with EC2

Add AmazonRDSFullAccess and AmazonDynamoDBFullAccess to the pizza-ec2-role

Automate Your App with Elastic Beanstalk and CloudFormation

Create, test, and delete pizza luvrs infrastructure with CloudFormation

  1. retrieve the pizza-plus-s3 AMI ID in the EC2 Dashboard AMIs section
  2. customize CloudFormation with your AMI ID
  3. Services, CloudFormation
  4. Create Stack
  5. upload the json template file
  6. name it pizza-stack then Create

Deploy an application with Elastic Beanstalk

  1. Services, Elastic Beanstalk
  2. Create New Application with name pizza luvrs
  3. Create Web Server
  4. select platform Node.js, and Load Balancing, Auto Scaling
  5. zip all the local code including dependencies with $ pizza-luvrs zip -r package.zip .
  6. setup Application Version, add the local zip file as Source, leave rest of the settings as it is
  7. check Create This Environment inside a VPC in the Additional Resources
  8. for Configuration Details, select pizza-keys as EC2 Key Pair, set Application Health Check URL as /, leave rest of the settings as it is
  9. for VPC Configuration, select the correct VPC ID, check both ELB and EC2 for the available Availability Zone, select pizza-ec2-sg as VPC Security Group
  10. select pizza-ec2-role for Instance Profile in Permissions setting page
  11. Launch

Configuring an Elastic Beanstalk environment

  1. Elastic Beanstalk Dashboard, pizza luvrs
  2. Configuration, Software Configuration, update the Node version to the latest version supported (6.2.2), Apply