Create health checks for main website

[Rabadash8820]

This is unrelated to #33, which was about the uneducated idea of "health checking DNS records", which is not a thing. Here we're talking about adding actual health checks to the website at www.derploid.com, so that we can be notified if/when the site is down. Additional health checks for the other, "redirect" domains might still be good too, so that we know all of those rules are working correctly.

[Rabadash8820]

So yeah, after giving this some more thought, this is how it should all work:

• Remember that there are:
  • 2 test "redirect" domains: derploidtest.click, www.derploidtest.click
  • 4 prod "redirect" domains: derploid.net, www.derploid.net, derploid.org, www.derploid.org
  • 1 test "main redirect" domain on the main test apex domain: derploidtest.link
  • 1 prod "main redirect" domain on the main prod apex domain: derploid.com
  • 1 test "main" domain: www.derploidtest.link
  • 1 prod "main" domain: www.derploid.com
• Define health checks:
  • One Route53 health check per domain (10 * $0.50 = $5.00 total)
  • If all health checks validate HTTPS, that's another 10 $1 = $10. Making the redirect domains use HTTP health checks would be cheaper, but then they would never fail (unless CF itself fails) because HTTP requests to the redirect domains are always redirected to HTTPS by CF, thus returning a "successful" 301 code. Really, the redirect Distribution should allow HTTPS or HTTP traffic, since the origin request will redirect everything to the main website over HTTPS anyway. HTTP health checks will still never fail though, b/c we have CF set up to redirect these domains to the main website domain, so again they'll always get a "successful" 301. We still need these health checks, but now, in a way, they're just checking the health of CF; only if the Distribution breaks in some way will the checks not get a 301. The main and main redirect domains have content hosted by GitHub Pages, using certs that we don't control, so we should still do HTTPS checks on them, for 4 $1 = $4.
  • Only health checks for the "main" and "main redirect" domains will measure latency, for 4 *$1 = $4. We don't really care if the other redirect domains are slow, but these domains should both be pretty quick (e.g., many people will type derploid.com directly into browsers, not realizing that it redirects to the www subdomain). And the test domains just have the checks so that the test environment stays as similar as possible to prod.
  • Only the main domains will validate the actual response, for 2 * $1 = $2. After all, if the main domain in test or prod is returning the wrong response, then all the other (main) redirect domains in that environment will be too (after CloudFront caches expire). We'll verify the HTML <title> of the home page, as this verifies that we didn't get an error page, and that the Jekyll builds worked correctly.
  • All told, these health checks should cost $5 + $4 + $4 + $2 = $15 per month. Not bad, not great.
• Define metrics/alarms
  • Each health check reports a HealthCheckStatus metric (among other metrics), so each will need its own separate CloudWatch Metric in CDK, with its own separate CloudWatch Alarm (since each domain could be separately misconfigured).
  • Only the main domain HealthCheckStatus alarm in each env will have an action: notify an SNS topic with configured emails, for the case when the main domain is unhealthy but the other domains are still good due to caching. This will alert us to act before all domains are unhealthy.
  • We will then add two composite alarms per environment:
    1. A UnhealthyWebsiteBreakingRedirects alarm with the rule "main domain unhealthy AND any of the (main) redirect domains unhealthy". This will spare us some alarm noise when the main website goes down: later, when the redirect domain caches expire and they start turning unhealthy, we won't get a new alarm for each domain.
    2. A RedirectDomainUnhealthy alarm with rule "main domain healthy AND any of the (main) redirect domains unhealthy". This will cause the same topic to be triggered by the same alarm anytime a (main) redirect domain is unhealthy
  • Add CloudWatch metrics and alarms for latency from the main (redirect) domain health checks. Main redirect should have slightly higher threshold than main to account for origin requests during cache misses. Neither alarm in each env should have an action.
  • Define a composite WebsiteLatencyThresholdExceeded alarm with rule "main domain latency OR main redirect domain latency are in alarm". This will cause the same topic to be triggered by the same alarm anytime a main (redirect) domain is taking too long.
• The list of emails to notify on health check alarms is separately configurable from the list used for DMARC reports

FWIW, I also considered these alternatives to Route53 HealthChecks:

1. CloudWatch Synthetic Canaries: these are automated canaries that call the configured endpoint on a regular schedule. They're very similar to Route53 HealthChecks, but able to validate multiple things per request, or even across multiple requests in a workflow (e.g., for automated UI testing). The additional power comes at a price: the first 100 canary runs per month are free, then its $0.0012 per canary run thereafter. Using the Pricing docs example of an endpoint called every 5 minutes, this would come about to another ~$10. We would probably check the "redirect" domains even less frequently, say once per hour or once per day, but across all (sub)domain checks, this could get pricey. Route53 run more often (every 30 sec max) for a lower price ($0.50 per health check per month + $1 per "optional" feature per month).
2. Defining Lambda Functions to call our endpoints regularly. This would give us the most power, and might even be the cheapest option, but would also require the most complexity and tedium to set up. Each Lambda Function would need to call an endpoint, check the HTTP status code, check the response body, parse it for particular strings, and send metrics to CloudWatch. This would also involve using CDK assets for the Lambda code, which I'm not really ready to learn about just for this use case, and things would get even more complicated to measure latency and other metrics that Route53 health checks already provide out of the box.