search

RobustPerception / azure_metrics_exporter

Azure metrics exporter for Prometheus
Apache License 2.0
133 stars 69 forks source link

Multidimensional metrics, multi subscriptions in a single config yml #76

Open mrajput84 opened 4 years ago

mrajput84 commented 4 years ago

Hi

As part of an ongoing effort to fetch Azure metrics for multiple providers (CosmosDB, VM scale sets, Web Sites etc), our team (walmart labs) has done a poc for this exporter. As per the initial analysis and feedback from customers, following are the main concerns:

  1. Multidimensional metrics are missing: For e.g. in case of CosmosDB, most of the available metrics have dimensions (AvailableStorage has e dimensions etc.). It seems there is no provision to fetch multi dimensional metrics as of this moment.

  2. In case we need to fetch metrics from multiple subscriptions, multiple instances of exporter need to be deployed (one for each subscription). Is there any specific reason that we can configure only one subscription in azure.yml.

  3. As per the current workflow all resource groups/resources need to be configured in yml file for which metrics need to be scraped (or tags). Since we have subscriptions which have more than 50 resource groups and multiple resources underneath, the azure yml file would be very tedious to write for all of the subscriptions.

  4. As per the current design, the call to fetch metrics and then publish all metrics to prometheus seems to be serially executed. Would it be better to parallelize the fetch and publishing part, any thoughts on this ?

We are working on an enhancement for multidimensional metrics and making the config file as simple as possible to configure. For e.g. simply specify the list of subscriptions with name and type of providers for which metrics need to be fetched and we’ll dynamically fetch resource groups/resources, fetch the available dimensions for metrics, formulate URLs and fetch all metrics. I will keep on updating this thread as and when we make progress. Please have a look at above mentioned points and share your thoughts.

brian-brazil commented 4 years ago

1: That sounds like a feature to be added.

2: This is similar to what the cloudwatch exporter does, you can only access one account at a time.

3: This sounds like something for configuration management, which I'd presume you already have at a scale of 50 things to manage. The goal is that the config format is comprehensible and writable by hand for someone starting out, not that it is as short as possible in complex use cases.

4: Azure has very low rate limits, and it's impolite to cause traffic spikes when you don't need to. Are you actually running into problems?

DenisBiondic commented 4 years ago

@mrajput84 "We are working on an enhancement for multidimensional metrics and making the config file as simple as possible to configure." we are also interested in this feature, did you already start with this? Do you need help?