Figure out how to collect AWS WAF logs using Amazon Data Firehose

[zmoog]

Goal

Suppose I own an AWS account, and I want to export AWS WAF events from AWS to an Elastic cluster.

Context

What are WAF (web application firewall) logs?

Requirements & Limitations

• supports EC (Elastic Cloud) deployments only
• EC deployment must be on AWS (deployments on Azure or GCP are not supported)
• does not support AWS PrivateLink yet (but it's coming soon)

Preparation

• install the latest AWS integration on your Elastic cluster

Steps

• Overview
• Select a resource
• Create a Firehose stream
• Create a Web ACL
• Set up logging
• Verify

[zmoog]

Overview

In this guide, we will select a WAF-compatible resource (CloudFront distribution) and create a web ACL to generate some WAF logs. Then we will edit the web ACL settings to forward all the logs to an Elastic stack using a Firehose stream.

[zmoog]

Select a resource

We can create and attach a web ACL to several AWS resources:

• CloudFront distribution
• Application Load Balancers
• Amazon API Gateway REST APIs
• Amazon App Runner services
• AWS AppSync GraphQL APIs
• Amazon Cognito user pools
• AWS Verified Access Instances

If you don't have an existing resource for testing, we suggest creating a simple CloudFront distribution.

[zmoog]

Create a Firehose stream

We need a Firehose stream to collect the WAF logs and send them to a data stream on an Elastic stack.

To create a Firehose stream, you can use the instructions at Monitor Amazon Web Services (AWS) with Amazon Data Firehose up to step 3. However, you must set two things differently.

• Name
• Parameters

Name

Your Firehose name must start with the the prefix aws-waf-logs- or it will not show up later.

Parameters

Follow the instructions up to step 3 except for the "parameters".

Use the following parameters:

Name	Value
es_datastream_name	logs-aws.waf-default

[zmoog]

Create a Web ACL

Visit the WAF & Shield page on the AWS console, and start creating a new web ACL.

At the time of the writing, creating a web ACL is a five steps process.

1. Describe web ACL and associate it to an AWS resource
2. Add rules and rule group
3. Set the rule priority
4. Configure metrics
5. Review and create web ACL

(1) Describe web ACL and associate it to an AWS resource

You we need to provide the following information:

• Resource type
• Name
• Associated AWS resource

(2) Add rules and rule group

Add a 1-2 rules in the "Free rule groups" list from the AWS managed rule groups.

Keep all other settings to their default values.

(3) Set the rule priority

Keep default values.

(4) Configure metrics

Keep default values.

(5) Review and create web ACL

In this final step, you can review your previous choices and confirm to create a web ACL.

[zmoog]

Set up logging

Visit the web ACL you create in the previous step.

Open the "logging and metrics" section and edit the settings.

• Logging destination: select "Amazon Data Firehose stream"
• Amazon Data Firehose stream: select the Firehose stream we created in a previous step.

WAF takes care of creating the needed IAM role for us.

If your Firehose stream name doesn't appear in the list, double-check its name starts with aws-waf-logs- as AWS prescribes.

[zmoog]

Verify

At this point, everything should be in place, and we can log into our Elastic stack to check if the WAF logs are flowing.

To generate logs, you can use cURL to send HTTP requests to our testing CloudFront distribution.

curl -i https://<your cloudfront distribution>.cloudfront.net

To maintain a steady flow of logs, you can use watch -n 5 to repeat the command every 5 seconds.

watch -n 5 curl -i https://<your cloudfront distribution>.cloudfront.net

Here are the first WAF logs from Firehose in Elastic.