Add support for custom metrics from Azure Monitor

[tomkerkhove]

Add support for custom metrics from Azure Monitor. These can originate from services such as Azure Application Insights.

Here is how you can add one.

Specification

• [ ] Add new scrape source
• [ ] Document new scrape source

[plooploops]

Would like to work on this one with Heba.

[tomkerkhove]

@plooploops Just out of curiosity - Did you manage to fix this? If not, what progress did you make and found any relevant docs?

If you prefer not to work on this anymore, no problem - Just let me know please!

[plooploops]

@tomkerkhove I can continue doing some research, but pulling custom metrics doesn't appear to be immediately available in the Azure Monitor SDK. We may be able to use the Resource URI template to pull a custom metric, but first we were having issues submitting to a REST endpoint for a storage account.

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-store-custom-rest-api

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/rest-api-walkthrough

We may need to wrap a REST API instead of using the SDK for now; I'll let you know how the research goes though.

[tomkerkhove]

@plooploops That's interesting - Maybe it already works then if we just specify the name of the custom metric on existing scrapers. Did you test that?

[jcorioland]

We are currently struggling with being able to get values from custom metrics. Our current case is with Azure Functions. When you create an Azure function, you have standard metrics available at the Microsoft.Web/sites level, but the Application Insights resource that is linked to the function also defines some interesting custom metrics (for each function in the function app):

In the picture above HttpTrigger1 is the name of my function.

After investigations, it seems that the Azure Monitor SDK (through the fluent API that you are using in Promitor) does not allow to retrieve those custom metrics. To be able to get them, you need to use the Application Insights SDK/REST APIs. I've done something like that:

var accesstoken = GetAccessToken("0000", "0000", "0000");
string url = "https://api.applicationinsights.io/v1/apps/APP_INSIGHTS_APP_ID/metrics/customMetrics/HttpTrigger1 Count";

var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Add("Authorization", "Bearer " + accesstoken);
var response = httpClient.GetAsync(url).Result;

if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
     string content = response.Content.ReadAsStringAsync().Result;
     Console.WriteLine(content);
}

That's being said, I think that to solve this issue, we could:

1. Implement a new integration provider for Application Insights (like for Azure Monitor and Storage)
2. Implement a new scraper for Application Insights (to get any of the standard or custom metrics)
3. Implement a new scraper for Azure Function that will use both Azure Monitor (for standard) and Azure Application Insights (for custom) integration providers

What do you think about? cc @tomkerkhove @brusMX

[jcorioland]

Also adding that it seems to have 2 kinds of custom metrics in Azure, one in Azure Monitor and one for Azure Application Insights. For the first one, I was not able to find a way to list them / retrieve their values from the Azure Monitor SDK. Looks like there is only one REST API to create new custom metrics ¯\(ツ)/¯

[tomkerkhove]

Thanks for the analysis @jcorioland and sorry for the late reply!

I've opened an issue related to Azure Functions metric duplication - https://github.com/Azure/azure-functions-host/issues/4746

In terms of the 2 approaches, I suggest to keep this one open for when Azure Monitor supports this and opened #645 to track the Application Insights approach

[dhagashtrivedi]

Hi @tomkerkhove, Has the support for scraping custom metrics from Azure Monitor been added?

[tomkerkhove]

We haven't made any change yet but I'd need to give it a go to see if it works out of the box.

Did you try it already?

[tomkerkhove]

Hi @tomkerkhove, Has the support for scraping custom metrics from Azure Monitor been added?

Can you elaborate on your scenario @dhagashtrivedi please? Do you write them to Azure Monitor or Application Insights?

[dhagashtrivedi]

Hi @tomkerkhove, yes we write them to Azure Monitor in a different metric Namespace.

[tomkerkhove]

Sounds good! Once it's out of preview I'll add this to our planning.