Figure out how to collect AWS VPC Flow logs using Amazon Data Firehose

[zmoog]

Goal

Suppose I own an AWS account, and I want to export AWS VPC Flow log events from AWS to an Elastic cluster.

Context

What are the VPC Flow logs?

Requirements & Limitations

• Supports Elastic Cloud (EC) deployments only
• The EC deployment must be on AWS (deployments on Azure or GCP are not supported yet)
• Does not support AWS PrivateLink yet (but it's coming soon)

Preparation

• Install the latest AWS integration on your Elastic cluster

Steps

• Overview
• Create a Firehose stream
• Create a flow log that publishes to Amazon Data Firehose
• Verify

Overview

Resources

• Publish flow logs to Amazon Data Firehose
• Create a flow log that publishes to Amazon Data Firehose

[zmoog]

Create a Firehose stream

We need a Firehose stream to collect the VPC Flow logs and send them to a data stream on an Elastic stack.

To create a Firehose stream, you can use the instructions at Monitor Amazon Web Services (AWS) with Amazon Data Firehose up to step 3.

However, you must set two things differently.

• Name
• Parameters

Name

Pick a name for your Firehose stream.

Parameters

Use the following parameters:

Name	Value
es_datastream_name	logs-aws.vpcflow-default

If you're publishing flow logs to a different account, create the required IAM roles, as described in IAM roles for cross account delivery.

[zmoog]

Create a flow log that publishes to Amazon Data Firehose

• Do one of the following:

  • Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/. In the navigation pane, choose Network Interfaces. Select the checkbox for the network interface.
  • Open the Amazon VPC console at https://console.aws.amazon.com/vpc/. In the navigation pane, choose Your VPCs. Select the checkbox for the VPC.
  • Open the Amazon VPC console at https://console.aws.amazon.com/vpc/. In the navigation pane, choose Subnets. Select the checkbox for the subnet.

• Choose Actions, Create flow log.

• For Filter, specify the type of traffic to log.

  • Accept – Log only accepted traffic
  • Reject – Log only rejected traffic
  • All – Log accepted and rejected traffic

• For Maximum aggregation interval, choose the maximum period of time during which a flow is captured and aggregated into one flow log record.

• For Destination, choose either of the following options:

  • Send to Amazon Data Firehose in the same account – The delivery stream and the resource to monitor are in the same account.
  • Send to Amazon Data Firehose in a different account – The delivery stream and the resource to monitor are in different accounts.

• For Amazon Data Firehose stream name, choose the delivery stream that you created.

• [Cross account delivery only] For IAM roles, specify the required roles (see IAM roles for cross account delivery).

• For Log record format, specify the format for the flow log record.

  • To use the default flow log record format, choose AWS default format.
  • To create a custom format, choose Custom format. For Log format, choose the fields to include in the flow log record.

• For Additional metadata, select if you want to include metadata from Amazon ECS in the log format.

• (Optional) Choose Add tag to apply tags to the flow log.

• Choose Create flow log.

[zmoog]

Verify

Now, the network interface / VPC / subnet you set up is sending the VPC flow logs to the Firehose data stream, which is forwarding them to the Elasticsearch cluster.

Here are the VPC flow logs in the logs-aws.vpcflow-default data stream using Discover and Log Explorer: