Figure out how to collect CloudTrail logs using the Elastic Agent (aws-s3 input)

[zmoog]

I want to collect Cloudtrail logs from a bucket named aws-cloudtrail-logs-1234-2761c2fa using the Elastic Agent, and send them to Elasticsearch.

[zmoog]

Overview

Here is the overview of the solution.

Architecture

┌─────────────────┐  (1) sends      ┌─────────────────┐               (4)
│   CloudTrail    │────events──────▶│    S3 bucket    │◀────────────fetches ──────────────┐
└─────────────────┘                 └─────────────────┘             new logs              │
                                             │                                            │
                                         (2) send                                         │
                                       notifications                          ┌ ─ ─ ─ ─ ─ ┼ ─ ─ ─ ─ ─
                                             │                                            │          │
                                             ▼                                │  ┌─────────────────┐
                                    ┌─────────────────┐      (3) fetches         │     aws-s3      │ │ (5) sends    ┌─────────────────┐
                                    │    SQS queue    │◀─────────new ─────────┼──│    <<input>>    │──────logs─────▶│  Elasticsearch  │
                                    └─────────────────┘     notifications        └─────────────────┘ │              └─────────────────┘
                                                                              │
                                                                               ─Elastic Agent─ ─ ─ ─ ┘

1. CloudTrail sends audit events to an S3 bucket in .json.gz files.
2. On each file creation, S3 sends a notification to the SQS queue
3. The Elastic Agent receives the notification
4. The Elastic Agent fetches the S3 object using the information from the notification.
5. The Elastic Agent sends the log events to Elasticsearch

Steps

Here are the required steps to implement the solution.

• Create a trail to export events to an S3 bucket.
• Create an SQS queue
• Allow the S3 service to publish message to the SQS queue
• Set up the S3 bucket to send notifications to an SQS queue.

[zmoog]

Create a trail to export events to an S3 bucket

Visit the AWS web console and create a trail, storing the trail the event in a S3 bucket.

In this example, we create a trail named my-trail that stores events in a S3 bucket named aws-cloudtrail-logs-1234-2761c2fa.

[zmoog]

Create an SQS queue

We need an SQS queue where we will send the S3 object creation notifications for the aws-cloudtrail-logs-1234-2761c2fa bucket.

Create a new SQS queue named mbranca-cloudtrail-logs-notifications-queue.

$ aws sqs create-queue --queue-name mbranca-cloudtrail-logs-notifications-queue
{
    "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/123/mbranca-cloudtrail-logs-notifications-queue"
}

[zmoog]

Allow the S3 service to publish message to the SQS queue

Allow S3 to send object creation notifications from aws-cloudtrail-logs-1234-2761c2fa to mbranca-cloudtrail-logs-notifications-queue:

$ cat policy.json
{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123:root"
      },
      "Action": "SQS:*",
      "Resource": "aws-cloudtrail-logs-123-2761c2fa"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:eu-west-1:123:mbranca-cloudtrail-logs-notifications-queue"
    }
  ]
}

# Set the SQS access policy
- Visit Amazon SQS > Queues > mbranca-cloudtrail-logs-notifications-queue > Access policy > Access policy (Permissions) 
- Edit, paste the content of `policy.json`, and save.

[zmoog]

Set up the S3 bucket to send notification to an SQS queue

The S3 bucket will notify the SQS queue whenever any s3:ObjectCreated:* event occurs.

# Enable notifications
$ cat notifications.json
{
    "QueueConfigurations": [
        {
            "Id": "Creations",
            "QueueArn": "arn:aws:sqs:eu-west-1:123:mbranca-cloudtrail-logs-notifications-queue",
            "Events": [
                "s3:ObjectCreated:*"
            ],
            "Filter": {
                "Key": {
                    "FilterRules": [
                        {
                            "Name": "Prefix",
                            "Value": ""
                        },
                        {
                            "Name": "Suffix",
                            "Value": ""
                        }
                    ]
                }
            }
        }
    ]
}

aws s3api put-bucket-notification-configuration \
    --bucket aws-cloudtrail-logs-1234-2761c2fa \
    --notification-configuration file://notifications.json

[zmoog]

Checkpoint

After this step, visit the AWS console and click on "Poll for messages" and you should be able to see a few messages with notification sent from S3.

Here's a sample notification:

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "aws:s3",
      "awsRegion": "eu-west-1",
      "eventTime": "2024-03-03T12:17:09.882Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "<redacted>"
      },
      "requestParameters": {
        "sourceIPAddress": "<redacted>"
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "Creations",
        "bucket": {
          "name": "aws-cloudtrail-logs-1234-2761c2fa",
          "ownerIdentity": {
            "principalId": "<redacted>"
          },
          "arn": "arn:aws:s3:::aws-cloudtrail-logs-1234-2761c2fa"
        },
        "object": {
          "key": "AWSLogs/<redacted>/CloudTrail/eu-west-1/2024/03/03/<redacted>_CloudTrail_eu-west-1_20240303T1215Z_wf92tb6UZ23lB814.json.gz",
          "size": 1206,
          "eTag": "f19ba3a4761c6598231c456bdb698d80",
          "sequencer": "0065E46A45D5F39177"
        }
      }
    }
  ]
}

[zmoog]

Set up the AWS integration

We must set up the integration to receive notifications from the SQS queue.

1. The "Collect logs via S3 Bucket" toggle must be off.
2. Set the SQS queue URL in the "[SQS] Queue URL" field.

After you assign the policy to an Agent, it will start collecting CloudTrails logs: