Alert on CloudTrail events from all regions across multiple accounts

[stoggi]

In a multiple account setup, I'd like to alert on CloudTrail events from all aws regions.

My current setup:

ProductionAccount

• CloudTrail enabled for all regions
• CloudWatch event rule to send all events to AlertingAccount Event Bus

AlertingAccount

• StreamAlert cluster in one region
• CloudTrail enabled for all regions
• CloudWatch event rule to send all events from ProductionAccount and AlertingAccount to prefix_cluster_stream_alert_kinesis

This works great for CloudTrail events that originate in the same region as the CloudWatch Event. But it doesn't work when the CloudTrail event occurs in a different aws region. It doesn't matter if the event comes from the ProductionAccount via the EventBus or from the AlertingAccount itself.

It seems CloudWatch events and rules are region specific. Additionally, I was unable to select the kinesis stream as a target of a CloudWatch event from a different region.

What is the best way to collect CloudTrail events from all regions from all accounts?

[stoggi]

After a bit of investigation I've come across the following options:

S3 bucket notifications

Enable CouldTrail for all regions in all accounts. Send the trails to a single s3 bucket in the StreamAlert cluster account/region. Use the s3:ObjectCreated:* notification to send the objects to the StreamAlert rule processor lambda.

Pros: Doesn't require infrastructure in all regions. Cons: 5-10 minute delay on events.

CloudWatch Logs

Enable CloudTrail for all regions in all accounts. Send the trails to a CloudWatch Logs destination in the StreamAlert cluster account/region. Add a subscription filter on the destination to send the events to the StreamAlert kinesis stream.

Pros: Doesn't require infrastructure in all regions. Cons: Events are gzipped, longer lambda processing time. 5-10 minute delay on events.

CloudWatch Events + EventBus

Enable EventBus in every region in every account to send CloudTrail events to the StreamAlert account. Create a CloudWatch Event in each region, to post to an SNS topic in each region. Use the SNS topic to post to a cross-region lambda function that sends them to the StreamAlert kinesis stream in the same region.

Pros: Minimal delay on events? Cons: Requires infrastructure in all regions.

(untested)

CloudTrail Processing Library

Run a Java application to poll the CloudTrail SQS stream using the CloudTrail Processing Library or Lambda function, and send the events to the kinesis stream. Not sure about cross account.

Pros: Minimal delay on events? Cons: Requires an ec2 instance, or Lambda to poll SQS.

(untested)

[stoggi]

I tried to add the kinesis stream as a target to the CloudWatch event rule from a different region, but without success:

aws --region us-east-1 events put-rule \
    --name "test_prod_streamalert_all_events" \
    --event-pattern "{\"account\": [\"ACCOUNT_ID\"]}" \
    --state "ENABLED" \
    --description "Capture all CloudWatch events" \
    --role-arn "arn:aws:iam::ACCOUNT_ID:role/streamalert/test_prod_streamalert_cloudwatch_role"

aws --region us-east-1 events put-targets \
    --rule "test_prod_streamalert_all_events" \
    --targets "Id"="StreamAlertKinesis","Arn"="arn:aws:kinesis:us-west-2:ACCOUNT_ID:stream/test_prod_stream_alert_kinesis"

Error:

An error occurred (AccessDeniedException) when calling the PutTargets operation: Access to the resource arn:aws:kinesis:us-west-2:ACCOUNT_ID:stream/test_prod_stream_alert_kinesis is denied. Reason: Adding cross-region target is not permitted.

[stoggi]

This means you do require the SNS topic and Lambda mentioned in CloudWatch Events + EventBus above.

Which would result in a structure similar to this:

[jacknagz]

Thanks for the analysis so far @stoggi!!

A couple notes on the approaches above:

CloudTrail Processing Library: Run a Java application to poll the CloudTrail SQS stream using the CloudTrail Processing Library or Lambda function, and send the events to the kinesis stream. Not sure about cross account.

This simply uses SQS to notify about delivery of new log files to S3. The StreamAlert rule processor is already equipped to analyze these logs

CloudWatch Events + EventBus: Enable EventBus in every region in every account to send CloudTrail events to the StreamAlert account. Create a CloudWatch Event in each region, to post to an SNS topic in each region. Use the SNS topic to post to a cross-region lambda function that sends them to the StreamAlert kinesis stream in the same region.

What's the purpose of the SNS topic in this scenario? The CloudTrail documentation suggests it's just for notifying for delivery of new log files to buckets. If we are creating CloudWatch events, we could send directly to Kinesis instead of SNS => Lambda => Kinesis.

It seems like we would run into the current issue of not being able to send cross region cloudwatch events to a single kinesis stream.

The options I see right now are:

1) Create a StreamAlert cluster in each region to accept events from within the same account and a cross-account event bus.

2) Configure S3 event notifications to Lambda to pull CloudTrail files directly from S3.

[stoggi]

That's a shame about the CloudTrail Processing Library. I had hoped you could get the CloudTrail events directly through SQS.

The SNS topic can be used to get around CloudWatch event rules not able to target cross-region resources. So the SNS Topic can notify a cross-region Lambda function which can post into the kinesis stream. But I agree it's too much.

Thanks for the feedback. I'll look at perhaps a combination of 1. and 2. Creating a StreamAlert cluster in each important region, and use s3 notifications for the rest.

[ghost]

hey @stoggi - just checking in to see if you got all squared away and if we can help in any way :)

[stoggi]

Hey @mime-frame, thanks for checking in. We settled on creating a StreamAlert cluster in each important region and using CloudTrail S3 notifications for the rest. I haven't had time to code it up yet, but I'm back on StreamAlert shortly so will see how it goes. Cheers.

[0xdabbad00]

I documented the approach I've used for this: https://summitroute.com/blog/2019/03/06/guardduty_event_collection_via_cloudwatch_events/