Reported values from filtered multi-dimensional metric are all the same

[DaveOHenry]

Report

As requested in https://github.com/tomkerkhove/promitor/discussions/1591

I tried to filter a multi-dimensional metric from Azure Application Gateway (see configuration below). Promitor returns the metric names as expected, but the values are the same for all instances (_1xx, _2xx, _3xx, _4xx, _5xx).

Unfortunately there seems to be another issue with the logging. As soon as I add the metrics configuration there are no Azure monitor logs anymore. Therefore I currently can't really figure out what the exact problem is.

fwiw: For my use-case it's currently enough to evaluate the "Total Requests" and "Failed Requests" metrics by backendpool. There is no filtering involved and it's possible to also use service discovery (I really like this feature! 😃).

Expected Behavior

Correct metric values for every instance.

Actual Behavior

Metric values for ... azure_application_gateway_backend_response_status_1xx azure_application_gateway_backend_response_status_2xx azure_application_gateway_backend_response_status_3xx azure_application_gateway_backend_response_status_4xx azure_application_gateway_backend_response_status_5xx ... are all the same.

Steps to Reproduce the Problem

1. add the scrape configuration
2. restart Promitor
3. let Prometheus scrape (or just call the metrics endpoint)
4. compare metric values ...

Component

Scraper

Version

2.1.0 (Helm Chart 2.2.1)

Configuration

Configuration:

metrics:
  - name: azure_application_gateway_backend_response_status_1xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '1xx'
  - name: azure_application_gateway_backend_response_status_2xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '2xx'
  - name: azure_application_gateway_backend_response_status_3xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '3xx'
  - name: azure_application_gateway_backend_response_status_4xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '4xx'
  - name: azure_application_gateway_backend_response_status_5xx
    description: "Http responses of Azure Application Gateway by backend and status code"
    resourceType: Generic
    azureMetricConfiguration:
      metricName: BackendResponseStatus
      aggregation:
        type: Total
      dimension:
        name: BackendPool
    resources:
      - resourceUri: Microsoft.Network/applicationGateways/my-appgw1234
        resourceGroupName: my-rg
        filter: HttpStatusGroup eq '5xx'

Logs

No response

Platform

Microsoft Azure

Contact Details

No response

[tomkerkhove]

Thanks for sharing, just for my understanding, would you mind adding a screenshot of Azure metrics explorer for these please?

[DaveOHenry]

[tomkerkhove]

Thank you!

[tomkerkhove]

Are you still seeing this?

[nikosmeds]

I believe I'm seeing the same issue. Adding details below, but let me know if you'd prefer a new issue instead @tomkerkhove.

Using image containers.promitor.io/tomkerkhove/promitor-agent-scraper:2.8.0.

Here's the metrics configuration. We're interested in splitting AllocatedSnatPorts by TCP and UDP, otherwise the result is the aggregate of the two.

    - azureMetricConfiguration:
        aggregation:
          type: Average
        dimension:
          name: BackendIPAddress
        metricName: AllocatedSnatPorts
      description: Total number of SNAT ports allocated within time period.
      name: azure_load_balancer_allocated_snat_ports_tcp
      resourceDiscoveryGroups:
      - filter: ProtocolType eq 'TCP'
        name: load-balancer-landscape
      resourceType: LoadBalancer
    - azureMetricConfiguration:
        aggregation:
          type: Average
        dimension:
          name: BackendIPAddress
        metricName: AllocatedSnatPorts
      description: Total number of SNAT ports allocated within time period.
      name: azure_load_balancer_allocated_snat_ports_udp
      resourceDiscoveryGroups:
      - filter: ProtocolType eq 'UDP'
        name: load-balancer-landscape
      resourceType: LoadBalancer

From Azure Monitor UI we see the result we're expecting (512, based on pool size):

However, Promitor is returning 1024 (the same value we see w/o the filter configured):

[00:10:13 INF] Found value 1024 for metric azure_load_balancer_allocated_snat_ports_tcp with dimension REDACTED_IP as part of BackendIPAddress dimension with aggregation interval 00:05:00
[00:10:13 INF] Found value 1024 for metric azure_load_balancer_allocated_snat_ports_tcp with dimension REDACTED_IP as part of BackendIPAddress dimension with aggregation interval 00:05:00
[00:10:13 INF] Found value 1024 for metric azure_load_balancer_allocated_snat_ports_tcp with dimension REDACTED_IP as part of BackendIPAddress dimension with aggregation interval 00:05:00
[00:10:13 INF] Found value 1024 for metric azure_load_balancer_allocated_snat_ports_tcp with dimension REDACTED_IP as part of BackendIPAddress dimension with aggregation interval 00:05:00
[00:10:13 INF] Found value 1024 for metric azure_load_balancer_allocated_snat_ports_tcp with dimension REDACTED_IP as part of BackendIPAddress dimension with aggregation interval 00:05:00
[00:10:13 INF] Found value 1024 for metric azure_load_balancer_allocated_snat_ports_tcp with dimension REDACTED_IP as part of BackendIPAddress dimension with aggregation interval 00:05:00
[00:10:13 INF] Found value 1024 for metric azure_load_balancer_allocated_snat_ports_tcp with dimension REDACTED_IP as part of BackendIPAddress dimension with aggregation interval 00:05:00

[tomkerkhove]

May I ask why you've added ProtocolType eq 'TCP' to resourceDiscoveryGroups? That is not really supported today.

[nikosmeds]

And that explains my issue. Filter is only available for Generic Azure Resource. Sorry for the noise!

[tomkerkhove]

No worries!