Figure out how the S3 request metrics integration works

[zmoog]

I want to enable the S3 request metrics integration to collect bucket metrics.

Using the S3 bucket named test-mbranca as a test case.

[zmoog]

Plan

1. Enable the request metrics (disabled by default) on the bucket
2. Send some requests to generate sample metrics
3. Check the metrics are visible on AWS
4. Collect the metrics using the Elastic Agent

[zmoog]

1. Enable the request metrics

Open the S3 bucket, select the Metrics tab, and click on View additional charts:

Then select the Request metrics tab, click on Manage filters, and Create filter:

Give the filter a name, a scope, and then click on Create filter:

[zmoog]

2. Send some requests to generate sample metrics

If your bucket is not currently used, you can send some requests using the AWS CLI.

Here's a trivial example:

watch -n 5 aws s3 ls s3://test-mbranca/

The watch command will execute the aws s3 ls s3://test-mbranca/ command every five seconds, so we will have some nice metrics for testing.

[zmoog]

3. Check if the metrics are visible on AWS

After a few minutes, request metrics should show up in the Metrics > Bucket Metrics page:

[zmoog]

4. Collect the metrics using the Elastic Agent

As last step, create a new agent policy, and install the AWS integration enabling the AWS S3 request metrics integration:

[zmoog]

The documents with the S3 request metrics should start showing up in Discover in a minute:

[zmoog]

Troubleshooting Guide

Here are some actions if you don't see the metrics in Discover.

Adjust latency

Sometimes metrics take some time to show up in CloudWatch, causing the Agent to miss them; to compensate this beheviour, we can use the latency setting in the advanced settings.

For example, here's how to add a two minutes latency:

With a 2m latency, the Agent will adjust the collection time windows by two minutes.