Not able to scrape WAF metrics

[akash123-eng]

I am not able to fetch waf metrics tried different config combination still not working

below is the one config.yml

# This is the default configuration for prometheus-cloudwatch-exporter
region: us-east-1
period_seconds: 240
metrics:
- aws_namespace: AWS/WAFV2
  aws_metric_name: AllowedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]
- aws_namespace: AWS/WAFV2
  aws_metric_name: BlockedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]
- aws_namespace: AWS/WAFV2
  aws_metric_name: CountedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]

I even tried setting Region parameter to global and then to us-east-1 i.e region where waf is present

# This is the default configuration for prometheus-cloudwatch-exporter
region: us-east-1
period_seconds: 240
metrics:
- aws_namespace: AWS/WAFV2
  aws_metric_name: AllowedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]
  aws_dimensions_select:
    Region: [Global]
- aws_namespace: AWS/WAFV2
  aws_metric_name: BlockedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]
  aws_dimensions_select:
    Region: [Global]
- aws_namespace: AWS/WAFV2
  aws_metric_name: CountedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]
  aws_dimensions_select:
    Region: [Global]

# This is the default configuration for prometheus-cloudwatch-exporter
region: us-east-1
period_seconds: 240
metrics:
- aws_namespace: AWS/WAFV2
  aws_metric_name: AllowedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]
  aws_dimensions_select:
    Region: [us-east-1]
- aws_namespace: AWS/WAFV2
  aws_metric_name: BlockedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]
  aws_dimensions_select:
    Region: [us-east-1]
- aws_namespace: AWS/WAFV2
  aws_metric_name: CountedRequests
  aws_statistics: [Sum]
  aws_dimensions: [WebACL, Rule, Region]
  aws_dimensions_select:
    Region: [us-east-1]

for all configs i am just getting cloudwatch requests for waf and not any waf metrics

nobody@global-prometheus-cloudwatch-exporter-6f88f79d86-p7997:/$ curl localhost:9106/metrics -s | grep -i waf
cloudwatch_requests_total{action="listMetrics",namespace="AWS/WAFV2",} 6.0
cloudwatch_requests_created{action="listMetrics",namespace="AWS/WAFV2",} 1.665649394337E9

please let me know how to fix this

[matthiasr]

Curious, unfortunately I don't have a WAF to test with. I noticed this in the documentation on the dimensions:

Region: Required for all protected resource types except for Amazon CloudFront distributions

Try removing the Region dimension from your configuration?

[akash123-eng]

@matthiasr forgot to reply.. I have already tried without region dimension and it still didn't work.. can you please check what might be issue ?

[akash123-eng]

@matthiasr please update on above

[matthiasr]

Unfortunately I can't really help you. There is nothing specific to WAF or any other of the hundreds of AWS services in the exporter. What you can do is explore the CloudWatch metric space in your account using the AWS CLI, to see what combinations of metric name, dimensions, and aggregations exist. If you can produce a CLI example of retrieving the metrics that you want in Prometheus, someone might be able to help you formulate the exporter configuration.

[peterpakos]

I'm also unable to scrape AWS/WAFV2 metrics using the following config:

  - aws_dimensions:
    - Region
    - Rule
    - WebACL
    aws_metric_name: AllowedRequests
    aws_namespace: AWS/WAFV2
    aws_statistics:
    - Sum
    aws_tag_select:
      tag_selections:
        Env: ["global"]
      resource_type_selection: "wafv2:regional/webacl"
      resource_id_dimension: WebACL
  - aws_dimensions:
    - Region
    - Rule
    - WebACL
    aws_metric_name: BlockedRequests
    aws_namespace: AWS/WAFV2
    aws_statistics:
    - Sum
    aws_tag_select:
      tag_selections:
        Env: ["global"]
      resource_type_selection: "wafv2:regional/webacl"
      resource_id_dimension: WebACL
  - aws_dimensions:
    - Region
    - Rule
    - WebACL
    aws_metric_name: CountedRequests
    aws_namespace: AWS/WAFV2
    aws_statistics:
    - Sum
    aws_tag_select:
      tag_selections:
        Env: ["global"]
      resource_type_selection: "wafv2:regional/webacl"
      resource_id_dimension: WebACL

Any idea how to fix this?

[peterpakos]

Without aws_tag_select it works just fine:

  - aws_dimensions:
    - Region
    - Rule
    - WebACL
    aws_metric_name: AllowedRequests
    aws_namespace: AWS/WAFV2
    aws_statistics:
    - Sum
  - aws_dimensions:
    - Region
    - Rule
    - WebACL
    aws_metric_name: BlockedRequests
    aws_namespace: AWS/WAFV2
    aws_statistics:
    - Sum
  - aws_dimensions:
    - Region
    - Rule
    - WebACL
    aws_metric_name: CountedRequests
    aws_namespace: AWS/WAFV2
    aws_statistics:
    - Sum

Any idea how to make it work with tags?

[matthiasr]

Interesting that this config works for you @peterpakos even though (in the parts you showed) it is the same as @akash123-eng's original one. You didn't include the global settings, how do they differ from the config in the description?

[matthiasr]

Carrying over from #475:

The Env tag is set for WebACL:

aws wafv2 list-tags-for-resource --resource-arn arn:aws:wafv2:eu-west-2:xxx:regional/webacl/yyy/zzz

{
    "NextMarker": "",
    "TagInfoForResource": {
        "ResourceARN": "arn:aws:wafv2:eu-west-2:xxx:regional/webacl/yyy/zzz",
        "TagList": [
            {
                "Key": "Env",
                "Value": "uk"
            },
            {
                "Key": "Environment",
                "Value": "Production UK"
            },
            {
                "Key": "ManagedBy",
                "Value": "Terraform"
            }
        ]
    }
}

[matthiasr]

It is actually quite involved for the exporter to get from the tag selection to the metric dimensions:

1. Get resources for tags using the generic resource groups tagging API
2. Extract the resource ID from the ARN
3. fetch metrics
4. filter the metrics based on the resource IDs extracted previously

I suspect something goes wrong along those lines – either the resource groups tagging API returns something unexpected, the ARN extraction fails, or the matching of the resource IDs to metric dimensions is off. Could you try to follow the same steps using the AWS CLI? Note that, because the exporter is generic, at no point should you encounter the aws wafv2 subcommand.

The goal here is to find out what is different about WAFv2. As a next step, we can then figure out how to handle this difference in the exporter.

[peterpakos]

Interesting that this config works for you @peterpakos even though (in the parts you showed) it is the same as @akash123-eng's original one. You didn't include the global settings, how do they differ from the config in the description?

Below are my global settings differing from the OP's original message:

region: eu-west-2
role_arn: arn:aws:iam::xxx:role/grafana
delay_seconds: 60
use_get_metric_data: true

I also found that using use_get_metric_data brings down the scrape duration from about 50s to 20s hence why I stick to it.

[omerlh]

I am having similar problem, I tried setting use_get_metric_data: true like @peterpakos did but no dice, still not WAF metrics. Anyone got it sorted?