fluent-plugin-ec2-metadata

Fluentd plugin to add Amazon EC2 metadata fields to a event record

Requirements

Installation

Use RubyGems:

gem install fluent-plugin-ec2-metadata

Configuration

Example:

<match foo.**>
  @type ec2_metadata

  aws_key_id  YOUR_AWS_KEY_ID
  aws_sec_key YOUR_AWS_SECRET/KEY

  metadata_refresh_seconds 300 # Optional, default 300 seconds
  imdsv2 true                  # Optional, default false

  output_tag ${instance_id}.${tag}
  <record>
    hostname      ${tagset_name}
    instance_id   ${instance_id}
    instance_type ${instance_type}
    az            ${availability_zone}
    private_ip    ${private_ip}
    vpc_id        ${vpc_id}
    ami_id        ${image_id}
    account_id    ${account_id}
  </record>
</match>

Assume following input is coming:

foo.bar {"message":"hello ec2!"}

then output becomes as below (indented):

i-28b5ee77.foo.bar {
  "hostname"      : "web0001",
  "instance_id"   : "i-28b5ee77",
  "instance_type" : "m1.large",
  "az"            : "us-west-1b",
  "private_ip     : "10.21.34.200",
  "vpc_id"        : "vpc-25dab194",
  "account_id"    : "123456789",
  "image_id"      : "ami-123456",
  "message"       : "hello ec2!"
}

Or you can use filter version:

<filter foo.**>
  @type ec2_metadata

  aws_key_id  YOUR_AWS_KEY_ID      
  aws_sec_key YOUR_AWS_SECRET/KEY

  metadata_refresh_seconds 300 # Optional, default 300 seconds
  imdsv2 true                  # Optional, default false

  <record>
    hostname      ${tagset_name}
    instance_id   ${instance_id}
    instance_type ${instance_type}
    private_ip    ${private_ip}
    az            ${availability_zone}
    vpc_id        ${vpc_id}
    ami_id        ${image_id}
    account_id    ${account_id}
  </record>
</filter>

Placeholders

The following placeholders are always available:

• ${tag} input tag
• ${tag_parts} input tag splitted by '.'. you can use it like ${tag_parts[0]} or ${tag_parts[-1]}
• ${instance_id} instance id
• ${instance_type} instance type
• ${availability_zone} availability zone
• ${region} region
• ${private_ip} private ip
• ${mac} MAC address
• ${vpc_id} vpc id
• ${subnet_id} subnet id
• ${account_id} account id
• ${image_id} ami image id

The followings are available when you define aws_key_id and aws_sec_key(or define IAM Policy):

• ${tagset_xxx} EC2 tag (e.g. tagset_name is replaced by the value of Key = Name)

The following is an example for a minimal IAM policy needed to ReadOnlyAccess to EC2.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "elasticloadbalancing:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:ListMetrics",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:Describe*"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "autoscaling:Describe*",
      "Resource": "*"
    }
  ]
}

Refer to the AWS documentation for example policies. Using IAM roles with a properly configured IAM policy are preferred over embedding access keys on EC2 instances.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request