AWS certification

[changtimwu]

force myself refresh everything by scheduling an urgent exam(June/13).

all white papers

https://aws.amazon.com/whitepapers/

all reinvents

https://reinventvideos.com/

all FAQs

https://aws.amazon.com/faqs/

just follow official preparation steps

https://aws.amazon.com/certification/certification-prep/

[changtimwu]

AWS Certified Cloud Developer Associate

• official page: https://aws.amazon.com/tw/certification/certified-developer-associate/
• https://acloud.guru/forums/aws-certified-developer-associate/
• https://www.linkedin.com/pulse/most-efficient-way-study-aws-certifications-associate-jady-liu/
• http://g23988.blogspot.com/2017/05/awsaws-certified-developer-associate-dva.html

jayendrapatil

• http://jayendrapatil.com/aws-certified-developer-associate-exam-learning-path/

some test taker notes

• http://blog.ioconnectservices.com/2018/03/aws-certified-solution-architect.html

brain dump

http://free-braindumps.com/amazon-braindumps-free.html

[changtimwu]

AWS SWF

*(simple workflow framework): https://aws.amazon.com/tw/swf/

• I won't use it since it requires an EC2 instance. step function & lambda should be enough for me
• reference: https://www.linkedin.com/pulse/aws-simple-workflow-vs-step-functions-shekhar-londhe/.
• (reliability)[https://aws.amazon.com/swf/faqs/]. Assign exactly once, never duplicated

limits

• https://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dg-limits.html
• number of domains: 100
• Maximum open workflow executions – 10,000 per domain
• Maximum workflow execution time – 1 year
• Maximum workflow execution history size – 25,000 events

---

• AWS securtiy toke service
• AWS Data pipeline
• AWS glue

[changtimwu]

section 8 SQS

• There is FIFO queue(exactly arrival once) and standard queue(qt least once)
• not guarantee FIFO. Can be arrived multiple times in any order.
• max 256kb 64k chunk. charged as four messages.
• best work with EC2 auto scaling
• analogy long term polling as blocked dequeue the the default timeout is 20 seconds. It's designed to avoid busy waiting problem of short term polling.
• use ReceiveMessageWaitTimeSeconds to adjust default timeout (0~20 seconds)
• SQS fan out ->
  • multiple SQS queues(Qb, Qc) subscribe one single SQS queue(Qa).
  • once Qa dequeues the message will be enqueue into both Qb and Qc.
  • the pubsub mechanism is usually done with SNS.

Visibility

• default is 30s. max is 12 hours.
• the API used to change visibility is ChangeMessageVisibility
• visibilitTimeout means other consumers won't be notified for that specific time. That means the consumer excusively own this messages for a period of time and this can avoid sync problem that multiple consumers concurrently process the same message.
  see details If you can't finish message processing within the timeout please use ChangeMessageVisibility to extend the timeout. _When you receive a message from a queue and begin to process it, the visibility timeout for the queue may be insufficient (for example, you might need to process and delete a message). You can shorten or extend a message's visibility by specifying a new timeout value using the ChangeMessageVisibility action.)
• if you set it to 0, that means you don't care the message might be receivered by other consumers. Maybe it's a broadcast message you just want it be spreaded as instant as possible.

---

• each queue has its own URL

  sqs get-queue-url --queue-name fuckq
  {
  "QueueUrl": "https://queue.amazonaws.com/406995953077/fuckq"
  }

  try read FAQ https://aws.amazon.com/sqs/faqs/ You can configure the Amazon SQS message retention period to a value from 1 minute to 14 days. The default is 4 days. Once the message retention limit is reached, your messages are automatically deleted.

  limits

• unlimited number of queues
• 1M free tier requests
• max 12 hours visibility
• 3 days to confirm
• 14 days retention

[changtimwu]

section 9 SNS

• SNS push messages. SQS poll messages.
• faq: https://aws.amazon.com/sns/faqs/
• always successd, non-blocked. We can tell that AWS is push based instead of pull based.
  • email, emai/JSON
  • SMS
  • SNS
  • http/https
  • mobile push notfication aka application
• auto backed up to multiple AZ
• subscription will be expired if not confirmed within 3 days
• Messages can be customised for each protocol used

Message

a message is made of the three attributes(name/type/value) https://docs.aws.amazon.com/sns/latest/dg/SNSMessageAttributes.html Name, type, and value must not be empty or null. In addition, the message body should not be empty or nul

{
  "Type" : "Notification",
  "MessageId" : "63a3f6b6-d533-4a47-aef9-fcf5cf758c76",
  "TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
  "Subject" : "Testing publish to subscribed queues",
  "Message" : "Hello world!",
  "Timestamp" : "2012-03-29T05:12:16.901Z",
  "SignatureVersion" : "1",
  "Signature" : "EXAMPLEnTrFPa37tnVO0FF9Iau3MGzjlJLRfySEoWz4uZHSj6ycK4ph71Zmdv0NtJ4dC/El9FOGp3VuvchpaTraNHWhhq/OsN1HVz20zxmF9b88R8GtqjfKB5woZZmz87HiM6CYDTo3l7LMwFT4VU7ELtyaBBafhPTg9O5CnKkg=",
  "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem",
  "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:123456789012:MyTopic:c7fe3a54-ab0e-4ec2-88e0-db410a0f2bee"
}    

example notfication to http end point

POST / HTTP/1.1
x-amz-sns-message-type: SubscriptionConfirmation
x-amz-sns-message-id: 165545c9-2a5c-472c-8df2-7ff2be2b3b1b
x-amz-sns-topic-arn: arn:aws:sns:us-west-2:123456789012:MyTopic
Content-Length: 1336
Content-Type: text/plain; charset=UTF-8
Host: example.com
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent

{
  "Type" : "SubscriptionConfirmation",
  "MessageId" : "165545c9-2a5c-472c-8df2-7ff2be2b3b1b",
  "Token" : "2336412f37fb687f5d51e6e241d09c805a5a57b30d712f794cc5f6a988666d92768dd60a747ba6f3beb71854e285d6ad02428b09ceece29417f1f02d609c582afbacc99c583a916b9981dd2728f4ae6fdb82efd087cc3b7849e05798d2d2785c03b0879594eeac82c01f235d0e717736",
  "TopicArn" : "arn:aws:sns:us-west-2:123456789012:MyTopic",
  "Message" : "You have chosen to subscribe to the topic arn:aws:sns:us-west-2:123456789012:MyTopic.\nTo confirm the subscription, visit the SubscribeURL included in this message.",
  "SubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=ConfirmSubscription&TopicArn=arn:aws:sns:us-west-2:123456789012:MyTopic&Token=2336412f37fb687f5d51e6e241d09c805a5a57b30d712f794cc5f6a988666d92768dd60a747ba6f3beb71854e285d6ad02428b09ceece29417f1f02d609c582afbacc99c583a916b9981dd2728f4ae6fdb82efd087cc3b7849e05798d2d2785c03b0879594eeac82c01f235d0e717736",
  "Timestamp" : "2012-04-26T20:45:04.751Z",
  "SignatureVersion" : "1",
  "Signature" : "EXAMPLEpH+DcEwjAPg8O9mY8dReBSwksfg2S7WKQcikcNKWLQjwu6A4VbeS0QHVCkhRS7fUQvi2egU3N858fiTDN6bkkOxYDVrY0Ad8L10Hs3zH81mtnPk5uvvolIC1CXGu43obcgFxeL3khZl8IKvO61GWB6jI9b5+gLPoBc1Q=",
  "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-f3ecfb7224c7233fe7bb5f59f96de52f.pem"
  }

mobile push

setup

• Request Credentials from Mobile Platforms: developer 去申請
• 用 credential 去 Request Token from Mobile Platforms
• 把 credential & token 交給 SNS, 產生出 Create Platform Application Object
• 再向 SNS 申請 Create Platform Endpoint Object

publish a message

• Publish Message to Mobile Endpoint

limits

Limits and Restrictions Q: Are there limits to the number of topics or number of subscribers per topic?

• 10 million subscriptions per topic,
• 100,000 topics per account. To request a higher limit, please contact Support.
• max 256 chars topic name

[changtimwu]

section 10 SWF

• workers receive tasks, process tasks, and return the result.
• interaction between workers and deciders
• a task is assigned only once and never duplicated?
• SQS: message oriented API, SWF:task oriented API
  • long term job or human involved jobs
• domain: task type
• max workflow can be 1 year?

[changtimwu]

section 11 Beanstalk

• For developers inexperience on AWS infrastructure.
• You app <-> EB App
• You app <-> mutilple EB Apps.
• beanstalk's deployment heavy covered in devops pro exam
  • immutable deployment: always create instance for new deployment instead of overwriting the existed
• rolling update: change VM/VPC settings without downtime
• one application can have multiple environments( launch configuration) both application and configuration can be changed
• how to deployment update can be adjust
  • all instances
  • rolling: one instance or % of instances a time
  • immuntable update

[changtimwu]

section 12 CloudFormation

• FAQ
• an easy to manage/create a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.
• five sections
  • resource
  • input
  • mappings: 放動態條件, AMI in a region might be different.
  • output
  • parameter
• you can modify and update them after deploy in an controllable way.
• CF template is some kind of blueprint file. CF stack is what deployed according an CF template.
• CF template can be in JSON or YAML
• optional elements of a CF template are stack creation time
• use Fn:GetAtt to get output data, which is what only can be told after deploy ex. public IP

---

AWSTemplateFormatVersion: "version date"
Description:
  String
Metadata:
  template metadata
Parameters:
  set of parameters
Mappings:
  set of mappings
Conditions:
  set of conditions
Transform:
  set of transforms
Resources:
  set of resources
Outputs:
  set of outputs

only Resources is required

intrinsic functions

• references
• Fn:GetAttr, Fn::FindInMap, Ref, Fn::Select(this is 0 based)
• "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "32"]

CLI

• list-stacks or describe-stacks
• describe-stack-resources --stack-name <stack name>

[changtimwu]

section 14 Route53

• When compared with CNAME, alias resource record saves your time. When a name to public IP changed, alias record reflects the change immediately.
• ELB doesn't have a predefined/fixed IP address. you resolve them using DNS.

[changtimwu]

dynamodb

FAQ

stream

• continuous message notfiication everytime the db gets updated.
• messages(in a stream) are kept for 24 hours

index

• query on non-primary key attributes using Global Secondary Indexes and Local Secondary Indexes.

capacity formula

• number of read simontanesly * read size round to 4KB /4KB
• and /2 if eventual consistency

  10 read 1KB  --> 10*4KB/4KB /2 =5 units of read throughput
  10 read 6KB --> 10*8KB/4KB / 2 = 10 units of read througtput
  5 read 10KB -> 5*12KB/4KB / 2 = 7.5 ~ 8 units of  read throughput
  5 read 10KB and strong consistency -> 5*12KB/4KB = 15 units of read throughtput
  5 write 10KB -> 5*10KB = 50 units of write throughput 
  12 write 100KB -> 12*100KB = 1200 units of write throughput
  10 write 15.5KB -> 10 * 16KB = 160 unit of write 

  throughput exceed errors -> HTTP 400 ProvisionedThroughputExceedException

AssumedRoleWithWebIdentity

limits

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html

• 256 tables per region.
• single item size: 400KB
• You can define a maximum of 5 local secondary indexes and 5 global secondary indexes per table.
• The maximum size of any item collection is 10 GB.
• The smallest Reserved Capacity offering is 100 capacity units (reads or writes).
• BatchGetItem operation can retrieve a maximum of 100 items. The total size of all the items retrieved cannot exceed 16 MB.
• A single BatchWriteItem operation can contain up to 25 PutItem or DeleteItem requests. The total size of all the items written cannot exceed 16 MB.
• The result set from a Query is limited to 1 MB per call. You can use the LastEvaluatedKey from the query response to retrieve more results.
• The result set from a Scan is limited to 1 MB per call. You can use the LastEvaluatedKey from the scan response to retrieve more results.

Query

Query vs Scan

Scan

• ProjectionExpression: All or Selected Attributes
• ScanIndexForward: Ascending or Descending

API brief

• Official doc:
• Control Plane: CreateTable, DescribeTable, ListTables, UpdateTable, DeleteTable
• Data Plane: PutItem, GetItem, BatchPutItem, BatchGetItem, Query(composite primary key)
• [concurrent item] write(https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithItems.html)_For example, you might want a PutItem operation to succeed only if there is not already an item with the same primary key. Or you could prevent an UpdateItem operation from modifying an item if one of its attributes has a certain value._

[changtimwu]

VPC

entity and relation

• 一個 region 內可以有好幾個 VPC, 一個VPC可以有好幾個 subnet
• A region must have one and only one default VPC
• 一個 VPC 要有一個 route table, 每個subet 也要有一個 route table, 預設共用同一個
• route table 在 create 時就要指定 belong to which VPC
• A VPC must have one and only one main route table
• route table association 有分 explicitly 與 implicitly, 一開始會所有 subnet 共用同一個 route table 是 implicitly association. *一開始開好的這個 route table, 稱為 main route table, 每次 create new subnet 預設就是走這個 main route table
• 一個 VPC 要有一個 network ACL, 每個subet 也要有一個 network ACL, 預設共用同一個
• 在 console 內單獨看單筆 route table or network ACL, 有一個 tab 會標註 subnet association 狀況
• 你可以去 subnet explicitly associate route table, 選 edit 不修改 save, 去 route table看再也不會警告 The following subnets have not been explicitly associated with any route tables and are therefore associated with the main route table
• network ACL 比較像 L3/L4 filter, drop or allow, non-stateful
• security group 是 stateful firewall

route and subnet

• 一開始就開好的這個 main route table 有兩個特色
  • 開好 default route 到 internet gateway -> 都出得去, 也可以說都是 public subnet.
  • 開好 172.172.31.0.0/16 to local -> VPC內不同 AZ可以互訪, 所以這裡講的是錯的

Facts

• EC2 instance 是開在 subnet 內的, 不能直接開在 VPC 內
• 一定要 create subnet 給 EC2, subnet的 cidr 必須要在 VPC 之內.
• subnet 不會跨 AZ, 通常一個AZ 一個subnet, 但也可以多個, 可以把AZ想像成 segment
• 2013年後 EC2 就一定要搭 VPC 了 稱為 EC2-VPC, 之前開的 instance 叫做 EC-classic
• 請把這張看懂
  • 三個 subnet, 1:10.0.0.0/24, 2:10.0.1.0/24, 3: 10.0.2.0/24
  • 每個 subnet 一個 route table
  • 每個 subnet 都接到一個 router, 透過那個 router 到 region 內, router 一台or多台不是重點, 想像那是一台 L3 switch, subnet 是 VLAN
  • according to their route table,
  • subnet 1 is allowed to visit the Internet. It's known as a public subnet.
  • subnet 2 is only allowed visit subnets within the same VPC. It's known as a private subnet.
  • subnet 3 can only visit corporate network. It's known as a VPN-only subnet.

• instance 想要開在 public subnet, 一定要有一個 public IP address

• 一個沒用過region, 裡面會自動開好
  • VPC * 1
  • subnet * 3(看有幾個AZ)
  • routes table * 1
  • dhcp * 1
  • internet gateway * 1
• 那個唯一的 route table belong to the VPC and all subnet
• 如果很久以前就開過 instance 沒 stop, 才會看到多一個 default VPC

NAT

• NAT instance or NAT gateway
• NAT instance 要記得關掉 source/dest check

[changtimwu]

S3

• global 沒分 region
• FAQ

bucket name rules

between 3 and 63 characters long, and can contain only lower-case characters, numbers, periods, and dashes. Each label in the bucket name must start with a lowercase letter or number.The bucket name cannot contain underscores, end with a dash, have consecutive periods, or use dashes adjacent to periods.

limit

• single object 5TB
• single put 5GB --> more than 100mb please use multipart upload
• 100 buckets per account

  encryption

• customer-provided encryption keys (SSE-C), request header specify x-amz-server-side​-encryption​ options
• SSE-S3: Each object is encrypted with a unique key employing strong multi-factor encryption. As an additional safeguard, it encrypts the key itself with a master key that it regularly rotates.
• SSE-KMS: There are separate permissions for the use of an envelope key (that is, a key that protects your data's encryption key)
• SSE-C: You manage the encryption keys and Amazon S3 manages the encryption,

  acceleration

• 先丟到 edge 再forward 到 s3 可以快 30%
• try here
• 選 bucket -> properties -> transfer acceleration -> enable
• endpoint 抄下來 ex. notes-timwuapp-uploads.s3-accelerate.amazonaws.com
• 照這裡 設定 cli, 以後 aws s3 cp 就會變快

  storage gateway

• an VM image you can install on your data center/server. then the storage can be managed by AWS console.
• easy integration of local and cloud storage
• four types four representation
  • file gateway -> S3 object
  • volume gateway -> EBS volume
  • cached volume -> S3 object
  • Tape gateway -> S3/Galicer
• error code list

request per second

• https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html
• if constantly more than 100 rps, consider randomize prefix of object name.

consistency

• read after write consistency for new object
• eventual consistency for existed object update

exam

• S3 bucket names may only contain only lower case letters, periods, numbers, and dashes. Bucket names must not be formatted as an IP address, and they may not begin with a period.
• it support multi object delete in single api https://aws.amazon.com/blogs/aws/amazon-s3-multi-object-deletion/

[changtimwu]

EC2

• FIGHTDRMCPX: FPGA, IOPS(High Speed Storage), Graphics, HighSpeedDisk, Trivial(dirty cheap), Dense Storage, Ram, M(many, general purpose), Compute, P( machine learning, high end graphic), X(exessive large memory)

• EBS type:

  • general SSD: GP2,
  • provision SSD: IO1
  • throught put optimized HDD: ST1, cannot be boot volume
  • Cold HDD: SC1, lowest cost, cannot be boot volume

• EFS: elastic file system, can be shared between instance

• 那個 region 的 VPC都還沒設定過, 開 instance 時 configure 的 network & subnet 還是會出現 VPC default

• 選 subnet 其實就是選 zone

• security group is virtual firewall.

• any rule you apply to security group take effect immediately.

• a role can be attached/detached to an instance while it's running(you can see if the role is taking effect by running aws inside the instance)

• create 的時候UI強迫一定要選 network 與 subnet, 這時候network 選 VPC, subnet 選 VPC 那邊 create 的subnet, 後面又要求 configure security group, 這個 SG的 firewall rule 似乎可以跟 subnet 完全不相容. ?

• 如果在 Create LB inside 選 VPC

  tested

• https://docs.aws.amazon.com/cli/latest/reference/ec2/attach-volume.html

• list ami images -> describe-images

[changtimwu]

IAM

• user, group, role, policies
• user belongs to a group and a group has policies
• user 不一定要給 console access, 可以只給 access key

---

role 有可以理解成兩個簡單的基本

• 給誰用 Trusted entities: Choose the service that will use this role Select type of trusted entity
• 什麼事 Policies:

[changtimwu]

EC2 Autoscaling

• 其實是描述一些條件, 根據條件來 add/terminate EC2 instance
  • min,max,desired
• 大多設定跟 EC2 一樣.

[changtimwu]

to be a partner?

• https://aws.amazon.com/partners/technology/
• https://aws.amazon.com/partners/service-delivery/?nc1=h_ls

[changtimwu]

Query and Scan

Query 與 Scan 可以對

• table or
• index

primary key

一個 table 的 primary key 可以是

• simple: type hash(aka partition key)
• composite: 上面 + type range(aka sort key) optional
• must be unique index 也是一樣道理

global secondary index

• 可以以多個(最多5個), 可以隨時加. *eventually consistency
• 自有capacity 與base table獨立
• 只抓 item 內 projected attributes 當對 primary key 以外的 attribute 查詢, 想要加速, consider a table called GameScores *userid - partition key
• game title - sort key
• other attributes, TopScore, TopScoreDateTime, Wins, Losses 一定要用 user id + game title 下去 query,

想只用 game title查, 或是用其他 attribute, 就只能用很慢的 scan

secondary index could be made of partiton key and sort key

• GameTitle as partition
• TopScore as sort GameScoreIndex 就可以拿來查某遊戲的最高分

local secondary index

• 必須一開始create table 時就順便創建
• eventually consistency + strong consistency
• capacity 與 base table一樣, 寫入時會瓜分 base table capacity
• 直接用 base table 的 partition key + 自選 attributes + sort key
• 可以抓任何 attribute
• for example: 想查某個 user 最近一次創高分(無論遊戲)是什麼時候, 就可以用TopScoreDateTime 做 local secondary index

  others

• create table and secondary index 不能同時, 等一個開完再下一個, 不然會看到 LimitExceedsException

[changtimwu]

ELB

https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-configure-load-balancer.html

[changtimwu]

Elastic cache

• https://aws.amazon.com/elasticache/faqs/
• not server less

[changtimwu]

After test I took the new version(released in June 2018). I quite like changes this time. More new stuff(XRay, build/deploy pipeline). More related to practical development. More rely on experience instead of memorization.

• please read the official material(https://aws.amazon.com/certification/certified-developer-associate/)
• cloudwatch custom metric
• code services
  • codecommit -- source repo
  • codebuild -- CI
  • codedeploy -- CD
  • codestar -- ide, team, wiki, issue, dashboard
  • codepipeline -- define stages, coordinate actions between stages.
• AWS XRay
• AWS CloudTrail
• SAM https://github.com/awslabs/serverless-application-model
• DAX https://aws.amazon.com/tw/dynamodb/dax/( sub-millisecond latency requirement)
• Where to place the config file of ElasticBeansTalk Extension
• backoff https://docs.aws.amazon.com/general/latest/gr/api-retries.html

SQS

• the meaning of at least once
• dead letter queue

KMS

• KMS API: https://docs.aws.amazon.com/kms/latest/developerguide/programming-top.html
• KMS + CloudTrail: https://docs.aws.amazon.com/kms/latest/developerguide/logging-using-cloudtrail.html
• How describe the lambda resource in cloudformat

Lambda

• lambda functions need to split if local test OK and lambda instance is enough
• lambda online test has not see any thing
• trace lambda function with xray
• does lambda role need cloudwatch policy to output debug msg? yes, the role must specify explict actions

  "Action": [
              "logs:CreateLogGroup",
              "logs:CreateLogStream",
              "logs:PutLogEvents"
          ],

  Rick Huang's survey

  https://rickhw.github.io/2018/06/05/AWS/AWS-Certified-DVA-C01-Update/

[changtimwu]

• try launch stack to launch a cloudformation template https://aws.amazon.com/blogs/devops/construct-your-own-launch-stack-url/
• https://aws.amazon.com/serverless/serverlessrepo/

codestar/codepipeline define the following

• source -> codecommit
• build -> codebuild
• test
• deploy -> cloudformation(not codedeploy?)
• monitoring -> cloudwatch

practical question

• buildspec
• codedeploy pricing
• codedbuild pricing

[changtimwu]

deeplens and sagemaker https://www.youtube.com/watch?v=cTsCMTxEPHs https://aws.amazon.com/about-aws/whats-new/2018/06/aws-deeplens-tensorflow-caffe-mxnet-kinesis-video-streams-buy-now/