search

fluent-plugins-nursery / fluent-plugin-cloudwatch-logs

CloudWatch Logs Plugin for Fluentd
MIT License
201 stars 141 forks source link
cloudwatch-logs fluentd

readme

fluent-plugin-cloudwatch-logs

Gem Version

CloudWatch Logs Plugin for Fluentd

Requirements

fluent-plugin-cloudwatch-logs fluentd ruby
>= 0.8.0 >= 1.8.0 >= 2.4
>= 0.5.0 && < 0.8.0 >= 0.14.15 >= 2.1
<= 0.4.5 ~> 0.12.0 * >= 1.9

Installation

For Fluentd

gem install fluent-plugin-cloudwatch-logs

For td-agent

td-agent-gem install fluent-plugin-cloudwatch-logs

Preparation

Create IAM user with a policy like the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:*",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:*:*",
        "arn:aws:s3:::*"
      ]
    }
  ]
}

More restricted IAM policy for out_cloudwatch_logs is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:PutLogEvents",
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Also, more restricted IAM policy for in_cloudwatch_logs is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:GetLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Authentication

There are several methods to provide authentication credentials. Be aware that there are various tradeoffs for these methods, although most of these tradeoffs are highly dependent on the specific environment.

Environment

Set region and credentials via the environment:

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_ACCESS_KEY"

Note: For this to work persistently the environment will need to be set in the startup scripts or docker variables.

AWS Configuration

The plugin will look for the $HOME/.aws/config and $HOME/.aws/credentials for configuration information. To setup, as the fluentd user, run:

aws configure

Configuration Parameters

The authentication information can also be set

Example

Start fluentd:

fluentd -c example/fluentd.conf

Send sample log to CloudWatch Logs:

echo '{"hello":"world"}' | fluent-cat test.cloudwatch_logs.out

Fetch sample log from CloudWatch Logs:

# stdout
2014-07-17 00:28:02 +0900 test.cloudwatch_logs.in: {"hello":"world"}

Configuration

out_cloudwatch_logs

<match tag>
  @type cloudwatch_logs
  log_group_name log-group-name
  log_stream_name log-stream-name
  auto_create_stream true
  #message_keys key1,key2,key3,...
  #max_message_length 32768
  #use_tag_as_group false
  #use_tag_as_stream false
  #include_time_key true
  #localtime true
  #log_group_name_key group_name_key
  #log_stream_name_key stream_name_key
  #remove_log_group_name_key true
  #remove_log_stream_name_key true
  #put_log_events_retry_wait 1s
  #put_log_events_retry_limit 17
  #put_log_events_disable_retry_limit false
  #endpoint http://localhost:5000/
  #json_handler json
  #log_rejected_request true
  #<web_identity_credentials>
  #  role_arn          "#{ENV['AWS_ROLE_ARN']}"
  #  role_session_name ROLE_SESSION_NAME
  #  web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
  #</web_identity_credentials>
  #<format>
  #  @type ltsv
  #</format>
</match>

NOTE: retention_in_days requests additional IAM permission logs:PutRetentionPolicy for log_group. Please refer to the PutRetentionPolicy column in documentation for details.

in_cloudwatch_logs

<source>
  @type cloudwatch_logs
  tag cloudwatch.in
  log_group_name group
  #add_log_group_name true
  #log_group_name_key group_name_key
  #use_log_group_name_prefix true
  log_stream_name stream
  #use_log_stream_name_prefix true
  state_file /var/lib/fluent/group_stream.in.state
  #endpoint http://localhost:5000/
  #json_handler json
  # start_time "2020-03-01 00:00:00Z"
  # end_time "2020-04-30 15:00:00Z"
  # time_range_format "%Y-%m-%d %H:%M:%S%z"
  # Users can use `format` or `<parse>` directive to parse non-JSON CloudwatchLogs' log
  # format none # or csv, tsv, regexp etc.
  #<parse>
  # @type none # or csv, tsv, regexp etc.
  #</parse>
  #<storage>
  # @type local # or redis, memcached, etc.
  #</storage>
  #<web_identity_credentials>
  #  role_arn          "#{ENV['AWS_ROLE_ARN']}"
  #  role_session_name ROLE_SESSION_NAME
  #  web_identity_token_file "#{ENV['AWS_WEB_IDENTITY_TOKEN_FILE']}"
  #</web_identity_credentials>
</source>

Test

Set credentials:

$ export AWS_REGION=us-east-1
$ export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY"
$ export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY"

Run tests:

rake test

Or, If you do not want to use IAM roll or ENV(this is just like writing to configuration file) :

rake aws_key_id=YOUR_ACCESS_KEY aws_sec_key=YOUR_SECRET_KEY region=us-east-1 test

If you want to run the test suite against a mock server, set endpoint as below:

export endpoint='http://localhost:5000/'
rake test

Caution

If an event message exceeds API limit (256KB), the event will be discarded.

Cross-Account Operation

In order to have an instance of this plugin running in one AWS account to fetch logs from another account cross-account IAM authentication is required. Whilst this can be accomplished by configuring specific instances of the plugin manually with credentials for the source account in question this is not desirable for a number of reasons.

In this case IAM can be used to allow the fluentd instance in one account ("A") to ingest Cloudwatch logs from another ("B") via the following mechanic:

IAM Detail: Consuming Account "A"

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sts:*"
            ],
            "Resource": [
                "arn:aws:iam::ACCOUNT-B:role/fluentd"
            ]
        }
    ]
}

IAM Detail: Log Source Account "B"

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT-A:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:DescribeDestinations",
                "logs:DescribeExportTasks",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:DescribeMetricFilters",
                "logs:DescribeSubscriptionFilters",
                "logs:FilterLogEvents",
                "logs:GetLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:eu-west-1:ACCOUNT-B:log-group:LOG_GROUP_NAME_FOR_CONSUMPTION:*"
            ]
        }
    ]
}

Configuring the plugin for STS authentication

<source>
  @type cloudwatch_logs
  region us-east-1      # You must supply a region
  aws_use_sts true
  aws_sts_role_arn arn:aws:iam::ACCOUNT-B:role/fluentd
  log_group_name LOG_GROUP_NAME_FOR_CONSUMPTION
  log_stream_name SOME_PREFIX
  use_log_stream_name_prefix true
  state_file /path/to/state_file
  format /(?<message>.+)/
</source>

Using build-in placeholders, but they don't replace placeholders with actual values, why?

Built-in placeholders use buffer metadata when replacing placeholders with actual values. So, you should specify buffer attributes what you want to replace placeholders with.

Using ${tag} placeholders, you should specify tag attributes in buffer:

<buffer tag>
  @type memory
</buffer>

Using %Y%m%d placeholders, you should specify time attributes in buffer:

<buffer time>
  @type memory
  timekey 3600
</buffer>

In more detail, please refer to the officilal document for built-in placeholders.

TODO

Contributing

  1. Fork it ( https://github.com/[my-github-username]/fluent-plugin-cloudwatch-logs/fork )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request