prometheus / cloudwatch_exporter

Metrics exporter for Amazon AWS CloudWatch
Apache License 2.0
900 stars 325 forks source link

[metrics]: Metrics are no longer pulled when using aws_tag_select #713

Open lukayeh opened 1 month ago

lukayeh commented 1 month ago

Context information

When running Cloudwatch exporter with aws_tag_select enabled for services like RDS and Elasticache I get metrics returned, however when doing the same for MSK and MQ it does not return any metrics.

We know this works

    region: eu-central-1
    period_seconds: 60
    metrics:
    - aws_namespace: AWS/AmazonMQ
      aws_metric_name: MessageCount
      aws_dimensions:
        - Broker
      aws_statistics:
        - Sum

This does not:

    metrics:
    - aws_namespace: AWS/AmazonMQ
      aws_metric_name: MessageCount
      aws_dimensions:
        - Broker
      aws_statistics:
        - Sum
      aws_tag_select:
        tag_selections: 
          Project: []
        resource_type_selection: "mq:broker"
        resource_id_dimension: [Broker]

This also does not work ❌:

    region: eu-central-1
    period_seconds: 60
    metrics:
    - aws_namespace: AWS/AmazonMQ
      aws_metric_name: MessageCount
      aws_dimensions:
        - Broker
      aws_statistics:
        - Sum
      aws_tag_select:
        tag_selections:
          Project: ["legit_tag"]
        resource_type_selection: "mq:broker"
        resource_id_dimension: "Broker"

For Kafka I've tried this but this also refuses to retrieve metrics:

  - aws_namespace: AWS/Kafka
    aws_metric_name: CpuCreditUsage
    aws_dimensions:
      - Cluster Name
    aws_statistics:
      - Average
    aws_tag_select:
      tag_selections: 
          Project: []
      resource_type_selection: "kafka:cluster"
      resource_id_dimension: "Cluster Name"

Exporter logs ```log WARNING: (listDimensions) ignoring metric AWS/AmazonMQ:MessageCount due to dimensions mismatch ``` ### What do you expect to happen? Metrics return when using `aws_tag_select` ### What happened instead? No metrics are returned due to `dimensions mismatch`
matthiasr commented 1 month ago

The "usual suspect" when aws_tag_select doesn't work is a mismatch between the ARN format (which the exporter gets back when asking for tags) and the resource ID dimension. According to this page, the broker ARN ends in broker:MyBroker:b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9 while the exporter, by default, looks for anything after the last :. I am guessing that in this case, the Broker dimension would be MyBroker, so maybe try something like arn_resource_id_regexp: 'broker:([^:]*):[^:]*$'?

matthiasr commented 1 month ago

For MSK, I can't really find a page about the ARN format, but this page has some examples that suggest a similar problem.

For both, once you get it to work, please submit a PR for the examples.