Application Insights query for "HTTP Server 2xx Failed Requests" alert should return requests in results

[nathancrtr]

Description

Underlying the Training HTTP Server 2xx (requests where (failed requests * 100.00 / total requests) >= 1) PagerDuty alert is an Application Insights query that returns 1 if the failed request threshold has been breached:

Screen Shot 2022-07-25 at 1.34.08 PM.png

This is valuable as it triggers an important alert and lets engineers begin an investigation promptly. However, since the failing requests themselves are not accessible from the log results, engineers must separately search for the failures.

Acceptance criteria

• The failed 2xx requests in the numerator of the alert's ratio should appear as individual request entries in the returned log search results.

[alismx]

This has been as completed as possible. A comment in the alert definition outlines how to get the results.

https://github.com/CDCgov/prime-simplereport/blob/41dc26e99d99af52118837fd5551e26fbffb8efd/ops/services/alerts/app_service_metrics/main.tf#L92