airbnb / streamalert

StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
https://streamalert.io
Apache License 2.0
2.86k stars 332 forks source link

Create community rule for Trusted Advisor events #618

Open 0xdabbad00 opened 6 years ago

0xdabbad00 commented 6 years ago

By default, StreamAlert monitors all CloudWatch events, but does not alert on all of them. The AWS Trusted Advisor service supposedly sends events to CloudWatch events by default. The TrustedAdvisor service checks for things like opening Security Groups publicly to port 3306 (mysql), making S3 buckets public, and a number of other checks. Some of these are free, and for those with Business support plans, there are alerts for things like Access Keys being leaked to Github.

We should have a Community rule to detect these, similar to what we have for GuardDuty: https://github.com/airbnb/streamalert/blob/master/rules/community/guardduty/guard_duty_all.py

Examples of these events are here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#trusted-advisor-event-types

0xdabbad00 commented 6 years ago

I'm currently struggling to get things Trusted Advisor is checking for to show up in my CloudWatch events.

The rule to create is going to nearly identical to the GuardDuty matcher https://github.com/airbnb/streamalert/blob/e86fab056214456edcb9d2482a0d132f21beb605/matchers/matchers.py#L22

Except, we'll use:

@matcher
def trusted_advisor(record):
    return record['source'] == 'aws.trustedadvisor'
0xdabbad00 commented 6 years ago

I have suspicions that Trusted Advisor issues only flow to CloudWatch Events if you have a Business support plan. I suspect this because programmatic access to the support API is only provided with that plan, and not the default "Basic" plan that I have been testing with, so if the API is restricted, then perhaps they've restricted other functionality like this.

0xdabbad00 commented 6 years ago

I received confirmation from someone that Trusted Advisor -> CloudWatch Events is only available for Business support contracts, so I'll test this in another account.

0xdabbad00 commented 6 years ago

Trusted Advisor events only go to us-east-1. We were able to get this integration working, and then by visiting the page https://console.aws.amazon.com/trustedadvisor/home and hitting the refresh button it caused StreamAlert to send alerts. Unfortunately, this created hundreds of alerts before we turned it off, as Trusted Advisor was generating events for all sorts of cost savings of a few pennies on every resource in the account. So we need to set up filtering for this.

I also don't know how often Trusted Advisor scans for these on it's own without something needing to go to that page, so I need to investigate that.