search

cds-snc / notification-planning-core

Project planning for GC Notify Core Team
0 stars 0 forks source link

Convert AWS Budget Alert from email to Slack #274

Open ben851 opened 5 months ago

ben851 commented 5 months ago

Description

As a developer of notify I would like to be alerted when we are running over budget on AWS. Currently a few people get emails notifying them, but it would be better to have a global message sent to slack so that everyone can be aware.

WHY are we building?

Increased visibility into spending in AWS.

WHAT are we building?

Create an SNS Topic and do slack integration as per this https://docs.aws.amazon.com/cost-management/latest/userguide/sns-alert-chime.html

VALUE created by our solution

More accountability for AWS costing, and we can surface potential cost savings.

Acceptance Criteria

QA Steps

maxneuvians commented 5 months ago

Every account has internal SNS topic that should be able to handle budget notifications https://github.com/cds-snc/sre-bot/blob/main/app/server/event_handlers/aws.py#L16-L17 and pipe them into slack

Screenshot 2024-01-11 at 10 56 04 AM
CalvinRodo commented 5 months ago

This is a function that SRE takes care of.

We already have internal alert in the #internal-sre-alerts channel that let us know how much each BU spent to date from the beginning of the month.

Here is an example of one of the alerts.

image

We also flag any account that had an increase of more then 35% in spending over the last day.

At the moment it's generated per account using the business unit tag that's attached to the AWS Account, you can see the tags we set here: https://github.com/cds-snc/aft-account-request/blob/main/terraform/platform.tf#L104-L108

We can easily split it down to a greater granularity then just the business unit because we are also tracking the product and environment.

P0NDER0SA commented 3 months ago

opened PR for this one -- added the code to TF to handle the notifications for slack on non dev environments (and email for dev environments)

P0NDER0SA commented 3 months ago

Added the notifications for Dev as well and removed any email notifications (all slack now). But we're getting an issue with AWS:

Dear AWS Customer,  March 20, 2024
You requested that we notify you regarding the status of your <notify-global-budget> budget on the following SNS topic: <arn:aws:sns:ca-central-1:800095993820:alert-general>.
Unfortunately, we are unable to successfully publish to this SNS topic at this time. Please ensure that AWS Budgets has been added to the list of services that are allowed to publish to this SNS topic.
sastels commented 3 months ago

Looking into this error today

P0NDER0SA commented 3 months ago

Sylvia from SRE mentioned there's an alerting module written for similar uses. https://github.com/cds-snc/terraform-modules/tree/main/spend_notifier

we're gonna investigate setting this up to replace or enhance our work

P0NDER0SA commented 3 months ago

works on staging! got the alert today.

P0NDER0SA commented 3 months ago

spoke. to Sylvia to figure out how to get the new SRE module to trigger and we have a couple of tweaks to make to the webhooks

P0NDER0SA commented 3 months ago

webhooks are created (there's a slack bot /sre webhooks create or /sre webhooks list) and the alerting is verified.