Feature request: Scraping metrics from custom namespaces

[alex-tsbk]

Hi. I'm wondering if there is a way to take advantage of auto-discovery but still use custom namespaces for some of the metrics. It's not really clear to me if it is already supported, so I'm raising this question here.

The most common use-case is obtaining MemoryUtilization metrics, which is not the default EC2 metrics dimension.

I would like to have structure similar to this one when defining my metrics:

discovery:
  exportedTagsOnMetrics:
    ec2:
      - Name
  jobs:
  - type: ec2
    regions:
      - us-east-1
    searchTags:
      - Key: Application
        Value: .*
    awsDimensions:
      - InstanceId
    addCloudwatchTimestamp: true
    metrics:
      - name: CPUUtilization
        period: 60
        length: 120
        statistics:
          - Average
      - name: MemoryUtilization
        namespace: MyCustomNamespace
        period: 60
        length: 120
        statistics:
          - Average

So the expected behavior, in this case, would be not using AWS/EC2 namespace, but looking up metrics in my custom namespace: MyCustomNamespace

Regarding label, this can still use the existing pattern, where namespace could be converted to lowercase: aws_ec2_mycustomnamespace_memoryutilization_average or just mycustomnamespace_memoryutilization_average.

These are just my own thoughts and I would be really thankful for any implementation if this functionality is supported eventually.

Underlying aws cli commands looks like this:

aws cloudwatch get-metric-statistics --metric-name MemoryUtilization --start-time 2020-07-30T20:22:00Z --end-time 2020-07-30T20:27:00Z --period 300 --namespace MyCustomNamespace --statistics Average --region us-east-1 --dimensions Name=InstanceId,Value=i-0123456789012345

I would put a PR but I have 0 proficiency in Go language.

P.S. Amazing work & amazing plugin!

[pluvio-phile]

Hi, I wanted to leave a comment about this feature request as well. I recently tried this exporter and it is amazing how fast and efficient it is. Please consider prioritizing this feature request if it is possible since application related metrics go to custom namespaces in cloudwatch.

Thank you so much for the outstanding work!

[thomaspeitz]

Thanks a lot for your warm words folks,

I would like to avoid to put custom namespace logic into autodiscovery. As this logic is already to complex and I need to refactor a lot.

What I could think of (shorterm) would be the following. Not quite sure if it solves your problem as if your dimensions do only contain the instanceId and not the tag Name it would not help much as it would be hard to monitor it.

customMetrics:
  metrics:
    # Scrape all metrics in namespace
    - namespace:
        - MyService
    # Scrape one metric in a namespace
    - namespace:
        - MyService
      metrics:
        - PageViewCount
    # Scrape only one metric in a namespace and filter exported metrics by Dimension value
    - namespaces:
        - MyService
      metrics:
        - PageViewCount
      searchDimensions:
        - key: Environment
          value: staging

Whats your opinion about such a setup?

[alex-tsbk]

Hi. Yes, this would work. Typically good practice is to augment exported metrics with tag metadata, so it should work.

[thomaspeitz]

Perfect will try to implement it in the next release(s). Need to verify how much work it is first.

[marius-jakobi]

Hi, first of all thank you for this Cloudwatch Exporter. Is there any update on this topic? Would be awesome to have this feature available soon.

[marius-jakobi]

@thomaspeitz Hi Thomas, is there any plan to implement this in the near future? Unfortunately I am not familiar with Go :(

[marius-jakobi]

Hi @thomaspeitz, any updates on this?

[thomaspeitz]

Hey Marius,

sorry I am not having time for this currently. Currently this project is not generating any revenue and as a freelancer it is hard to give myself time for it.

Greetings :)