Redundant WAF blocked alarm when many rate blocked requests

[dsotirho-ucsc]

Currently there are two WAF blocked alarms, one (waf_rate_blocked) that is triggered when the rate (num per 5min) of rate-blocked requests exceeds a threshold, and one (waf_blocked) that is triggered when the percentage of requests blocked for any reason over the last 15 minutes exceeds a threshold.

Due to the overlap of coverage in these alarms, when the number of rate-blocked requests is high, both alarms can be triggered. This is redundant, and ideally only the waf_rate_blocked alarm would be tripped in this case.

Proposed solution is to modify the waf_blocked alarm so that it is based on the percentage of all blocked requests except the rate-blocked requests. Instead of using the metric_query dimension Rule: ALL, perhaps the Terraform expression for_each can be used to populate the rule dimension with the aws_wafv2_web_acl rule names.

https://github.com/DataBiosphere/azul/blob/7ea9749d71cefc3d923349a21607f2548daa6b00/terraform/cloudwatch.tf.json.template.py#L270

[dsotirho-ucsc]

Assignee to propose a more concrete solution.

[dsotirho-ucsc]

Assignee to determine if multiple dimensions with the same name can be specified in a metric query object.

[dsotirho-ucsc]

Assignee to determine if multiple dimensions with the same name can be specified in a metric query object.

Names must be unique.

$ aws cloudwatch put-metric-alarm \
--no-actions-enabled \
--alarm-name "dimension-test" \
--comparison-operator "GreaterThanThreshold" \
--threshold 1 \
--period 300 \
--evaluation-periods 1 \
--statistic "Sum" \
--namespace "AWS/WAFV2" \
--metric-name "BlockedRequests" \
--dimensions "[\
{\"Name\":\"WebACL\",\"Value\":\"azul-api_gateway-daniel\"},\
{\"Name\":\"Region\",\"Value\":\"us-east-1\"},\
{\"Name\":\"Rule\",\"Value\":\"BlockedIPs\"},\
{\"Name\":\"Rule\",\"Value\":\"AWS-CommonRuleSet\"}]"

An error occurred (ValidationError) when calling the PutMetricAlarm operation: Dimension names must be unique (duplicate: Rule)