Splitting metrics by source

[MostefaKamalLala]

Hello, I managed to make everything work in with my prometheus stack however I couldn't find how to split a metric like it's possible to do in grafana or Azure. Basically I want to split my node_cpu_usage_percentage by host. Is there something to add in params or a special way to write it in the filter? Thank you

[mblaschke]

What you're looking for is the dimension support: https://github.com/webdevops/azure-metrics-exporter#storageaccount-metric-namespace-and-dimension-support

[MostefaKamalLala]

Thank you @mblaschke I was misleaded by the the value in Azure UI which is host but in reality the dimension is node when I refered to the list of supported metrics following your guidance. Thank you very much!

[MostefaKamalLala]

I tried it out it's almost working but I found another issue, the result is different from Azure Monitor UI. I'm missing nodes from my second nodepool.

scrape_configs:
  - job_name: azure-metrics-example
    scrape_interval: 1m
    metrics_path: /probe/metrics/list
    params:
      name:
        - azure_metric
      template:
        - '{name}_{metric}_{aggregation}_{unit}'
      subscription:
        - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      resourceType:
        - Microsoft.ContainerService/managedClusters
      metric:
        - node_cpu_usage_percentage
      interval:
        - PT1H
      timespan:
        - PT1H
      aggregation:
        - average
      metricFilter:
        - node eq '*'
      metricTop:
        - 20
    static_configs:
      - targets:

I only get results from my linux vmss but i'm supposed to get it too from windows vmss.

[mblaschke]

Thank you @mblaschke I was misleaded by the the value in Azure UI which is host but in reality the dimension is node when I refered to the list of supported metrics following your guidance. Thank you very much!

one of the problem for Azure Monitor: the bad documentation and that Azure portal is using different terms :(

I only get results from my linux vmss but i'm supposed to get it too from windows vmss.

you're sharing the config example for keyvaults? 🤔

[MostefaKamalLala]

@mblaschke Updated. I Edited my code snippet and I found the issue. I was seeing the metrics on the vmss metrics namespace in the UI of azure which is different from the managedClusters metrics namespace. For some reasons, It's not returning all the nodes from the latest.

So I was able to reproduce the call with postman using azure metrics api however I found that the current apiversion that is used with this exporter is 2018-01-01 in the metrics client.

// ListPreparer prepares the List request.
func (client MetricsClient) ListPreparer(ctx context.Context, resourceURI string, timespan string, interval *string, metricnames string, aggregation string, top *int32, orderby string, filter string, resultType ResultType, metricnamespace string) (*http.Request, error) {
    pathParameters := map[string]interface{}{
        "resourceUri": resourceURI,
    }

    const APIVersion = "2018-01-01"
    queryParameters := map[string]interface{}{
        "api-version": APIVersion,
    }

Which give me this errror: "ApiVersion: 2018-01-01 does not support query at non Arm resource Id level"

I'll try to find a way to change to api version.

[MostefaKamalLala]

Got it working with your exporter finally. It was all about querying the right place. It was a pain finding it. Here is my final working configuration:

scrape_configs:
  - job_name: azure-metrics-example
    scrape_interval: 1m
    metrics_path: /probe/metrics/list
    params:
      name:
        - azure_metric
      template:
        - '{name}_{metric}_{aggregation}_{unit}'
      subscription:
        - xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
      resourceType:
        - Microsoft.ContainerService/managedClusters
      metricNamespace:
        - insights.container/nodes
      metric:
        - cpuUsagePercentage
      interval:
        - PT1H
      timespan:
        - PT1H
      aggregation:
        - average
      metricFilter:
        - host eq '*'
      metricTop:
        - 20
    static_configs:
      - targets:
          - url-to-your-azure-metrics-exporter-instance

[mblaschke]

Keep in mind that you now get the top most 20 entries for the metrics.

I've tried to keep the naming close to the Azure Monitor API but maybe not the best idea :(