Monitoring - Split PROD & STAGING `build-error-check` Canary into 2 Separate Canaries

omgitsbillryan commented 3 years ago

The Problem

The build-error-check AWS Synthetic Canary currently checks both staging.va.gov & va.gov. Each env should have it's own canary so they can have different alert rules & thresholds.

Extra Context

This could easily be done by using canary ENV variables - each canary would share the same JS code but have different ENV values. However, ENV vars for canaries are currently not supported in terraform. We could manually set these ENV vars in the AWS Canary UI, but they would get wiped each time the terraform for that canary gets applied.

The other alternative would be to duplicate the javascript code on our side - one for each canary. I opted against this and chose instead to just keep 1 canary that checks both envs.

Work to be Done

[ ] Have 2 separate canaries running, one for staging and one for prod
[ ] Each canary has its own CW alarm & SNS topic connections

jhouse-solvd commented 3 years ago

@pjhill & @JoeTice - This was just a suggestion based on where we left the canary when we shipped it to cover the base use cases that Dror was interested in. Can you take a look at this and see if it would still be relevant to the work that y'all are doing around synthetics?

jbritt1 commented 2 years ago

@pjhill & @JoeTice, since synthetic-style monitoring has mostly gone to Datadog, I would feel this ticket to no longer be relevant or a priority. Could we close in your opinion(s)?

jbritt1 commented 2 years ago

Bumping for feedback. Also posted in channel: https://dsva.slack.com/archives/C01CHAY3ULR/p1657739962454579

department-of-veterans-affairs / va.gov-team