Want to modify tags "keys" to lowercase scraping in Cloudwatch-Exporter before sending to Prometheus

[Vaibhav-1995]

We are using Cloudwatch-exporter to scrape cloudwatch metrics and adding exporter job in Prometheus to scrape those metrics from exporter and then forwarding those metrics from prometheus to mimir to enable alerting on metrics.

There is no consistency in keys of tags at aws side, due to which we are facing issue of getting blank values of tags in alert email as many of them are not matching keys in alert template code as we can add only one pattern (Uppercase or lowercase) in template code.

So we are trying to standardise the incoming tags from all aws services to one fix pattern as lowercase and then forward them to prometheus so this issue will be overcome and will able to print all tags in alert email.

Please suggest how we can achieve this with cloudwatch-exporter job scrapinng config that we have added as below -

  - aws_dimensions:
    - InstanceId
    aws_metric_name: CPUUtilization
    aws_namespace: AWS/EC2
    aws_statistics:
    - Average
    aws_tag_select:
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

tags we are getting through below metrics -

aws_resource_info{job="aws_ec2",instance="",tag_Budget_Code=XXX,tag_Name="XXX",tag_Owner="XXX",tag_businessunit="XXX",tag_environment="dev"} 1.0

So, here we want to modify tag_Budget_Code to tag_budget_code, tag_Name to tag_name, tag_Owner to tag_owner and so on all tags starting with word "tag_" . So need some dynamic config that will convert all tags starting with word "tag_" to lowercase.

Please suggest how we can do this with cloudwatch-exporter config as already raise it with Prometheus Community https://groups.google.com/g/prometheus-users/c/AOj-VHeRsGA and Prometheus can only convert label "values" to lowercase but not labels "keys".

[Vaibhav-1995]

We are using Cloudwatch-exporter to scrape cloudwatch metrics and adding exporter job in Prometheus to scrape those metrics from exporter and then forwarding those metrics from prometheus to mimir to enable alerting on metrics.

There is no consistency in keys of tags at aws side, due to which we are facing issue of getting blank values of tags in alert email as many of them are not matching keys in alert template code as we can add only one pattern (Uppercase or lowercase) in template code.

So we are trying to standardise the incoming tags from all aws services to one fix pattern as lowercase and then forward them to prometheus so this issue will be overcome and will able to print all tags in alert email.

Please suggest how we can achieve this with cloudwatch-exporter job scrapinng config that we have added as below -

  - aws_dimensions:
    - InstanceId
    aws_metric_name: CPUUtilization
    aws_namespace: AWS/EC2
    aws_statistics:
    - Average
    aws_tag_select:
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

tags we are getting through below metrics -

aws_resource_info{job="aws_ec2",instance="",tag_Budget_Code=XXX,tag_Name="XXX",tag_Owner="XXX",tag_businessunit="XXX",tag_environment="dev"} 1.0

So, here we want to modify tag_Budget_Code to tag_budget_code, tag_Name to tag_name, tag_Owner to tag_owner and so on all tags starting with word "tag_" . So need some dynamic config that will convert all tags starting with word "tag_" to lowercase.

Please suggest how we can do this with cloudwatch-exporter config as already raise it with Prometheus Community https://groups.google.com/g/prometheus-users/c/AOj-VHeRsGA and Prometheus can only convert label "values" to lowercase but not labels "keys".

Hi Team,

Any Update on this?

[matthiasr]

I don't want to add this feature to the exporter, because it breaks with both AWS and Prometheus behavior.

AWS documents that tags are case sensitive and you should

Use a standardized, case-sensitive format for tags, and apply it consistently across all resource types.

PromQL also treats labels as case sensitive%2C%20%22Label1%22%2C%20%22Upper%20Case%22%2C%20%22name%22%2C%20%22.*%22)&g0.tab=1&g0.display_mode=lines&g0.show_exemplars=0&g0.range_input=1h), so there is really no mismatch here. If the exporter were to squash tags to lower case, it would conflate tags that AWS and Prometheus both consider explicitly separate.

If you have to deal with specific labels that your organization applies inconsistently, the best path forward would be to fix these at the source. If this is not possible at all, and you absolutely need to deal with this, you can use metric relabeling to rename specific tags, like so:

# Copy the value between the different labels
- source_labels: [tag_wEird_CaSiNg]
   target_label: tag_weird_casing
# Clear the old label
- source_labels: [__name__]
  target_label: tag_wEird_CaSiNg
  replacement: ''

You will have to do this for each mis-capitalized tag though.