make logstash_input_script.js to Crawler more performance data from AWS CloudWatch

[uniray7]

EC2:

• CPUUtilization:
  • Average
  • Min
  • Max
• MemoryUtilization:
  • Average
  • Min
  • Max
• Disk:
  • DiskSpaceUtilization
  • Average
  • Min
  • Max
  • VolumeWriteOps
  • Average
  • Min
  • Max
  • VolumeWriteBytes
  • Average
  • Min
  • Max
  • VolumeReadOps
  • Average
  • Min
  • Max
  • VolumeReadBytes
  • Average
  • Min
  • Max

ELB:

• Latency
  • Average
  • Min
  • Max ELB其他的metrics還不了解是甚麼意思, 之後再看文件 http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-cloudwatch-metrics.html#loadbalancing-metrics

S3:

• NumberOfObjects
  • Average
• BucketSizeBytes
  • Average

[uniray7]

目前只先做EC2的部分 ELB 和S3之後再補

目前規劃的aws-performance schema如下: (另外還會用針對我們各種app的schema)

_index: aws-{YYYY.MM}

• 每個月建立一個index
• index就如同RDBMS裡的db, 可以指定分割成多個shard, 存成多個replica
• 必須利用insert index, 來指定你的index必須切幾個shard, 存幾個replicas, 指定每個field地的type是time, string..., 需不需要anlyze
• 但因為每個月建立一個新的index, 為了避免每個月都要重新設定, 可以使用import index template
• e.g:

"_index": "aws-2016.07"

_type: [ec2, elb, s3]

• 就如同RDBMS裡的table
• 目前只做ec2部分
• e.g:

"_type": "ec2"

_id:AVXd6w7U07peuaRbsSDy

• just document id

target: {instanceId, name, component}:

• 未來這schema可能會因為加入elb和s3而再改
• e.g.:

"target": {
    "instanceId": "i-0cf8e93907cfc63fc", 
    "name": "verpix-production-mongodb-secondary1", 
    "component": "mongodb"
}

metric: {Timestamp, Statistic, Value, Unit, type}

• Statistic: [Average, Maximum, Minimum]

"metric": {
    "Timestamp": "July 12th 2016, 14:57:00.000",
    "Statistic": "Average",
    "Value": 25.486,
    "Unit": "Percent",
    "Type": "MemoryUtilization"
}

host: ip-172-31-17-129

• just hostname of the vm

Summary:

所以整體下來json長得像這樣:

{
    "_index": "aws-2016.07",
    "_type": "ec2",
    "_id":"AVXd6w7U07peuaRbsSDy",
    "target": {
        "instanceId": "i-0cf8e93907cfc63fc", 
        "name": "verpix-production-mongodb-secondary1", 
        "component": "mongodb"
    },
    "metric": {
        "Timestamp": "July 12th 2016, 14:57:00.000",
        "Statistic": "Average",
        "Value": 25.486,
        "Unit": "Percent",
        "Type": "MemoryUtilization"
    },
    "host": "ip-172-31-17-129" 
}