search

hubverse-org / hubverse-cloud

Test hub for S3 data submission and storage
MIT License
0 stars 0 forks source link

Create AWS alert for unusual activity #17

Closed bsweger closed 6 months ago

bsweger commented 7 months ago

Probably doesn't need to be a high priority as we get started, but because we're hosting publicly-accessible S3 buckets, we should have AWS alert us to any spikes/high-volumes of data access activity (since we'll be paying for it).

bsweger commented 7 months ago

Enabled the billing and cost management console in the Hubverse AWS account (first step towards turning on cost anomaly detection).

AWS says that we have to wait 24 hours for the data to appear.

bsweger commented 6 months ago

Gonna try to set this up now because I'm curious to see if the upcoming flusight historical data onboarding will trigger off alerts about unusual spend/activity.

bsweger commented 6 months ago

Used [these directions](https://docs.aws.amazon.com/cost-management/latest/userguide/getting-started-ad.html to create a cost anomaly detection on the Hubverse AWS account's S3 service.

ARN of the detection: arn:aws:ce::767397675902:anomalymonitor/5567bc81-0cbe-4a97-8220-a988c44ad0531

The detection is configured to send an e-mail to me and @nickreich if S3 spend increases by 10% or more over the expected threshold (which is determined via machine learning ✨ )

Since we don't have good baseline data, there may be some false positives as we start moving large volumes of info to S3 (e.g., flu archive). It's straightforward to tune the threshold via the AWS console (and to add new alert recipients).