WebJobsTelemetryInitializer causes throttling of Application Insights custom metrics

olegsych commented 3 years ago

Default instrumentation of Azure Functions produces metrics with unbounded cardinality, which leads to throttling of custom metrics in Azure Monitor.

Repro steps

Enable integration with Application Insights and add the following code to your function.

var telemetry = new MetricTelemetry("MyNamespace, "MyMetric", count, sum, min, max);
telemetryClient.TrackMetric(telemetry);

Expected behavior

MyMetric is tracked without high-cardinality custom dimensions.

Actual behavior

MyMetric is tracked with the InvocationId dimension, an unbounded cardinality value, which at high enough volume get's the custom metrics throttled for the entire subscription.

Known workarounds

Override default functions configuration?

Related information

https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-custom-overview#quotas-and-limits

Package version (latest used by functions)
Links to source (n/a)

v-bbalaiagar commented 3 years ago

Hi @brettsam, Could you please suggest the repo to transfer this issue

brettsam commented 3 years ago

@v-bbalaiagar -- this is the correct repo.

v-bbalaiagar commented 3 years ago

Hi @brettsam, Can you suggest how to further proceed to triage this issue. Do we require more information for further investigation

v-bbalaiagar commented 3 years ago

Hi @olegsych, Can you please give us more information regarding the scenario to reproduce the issue.

olegsych commented 3 years ago

@v-bbalaiagar I have provided this information in the issue. Do you have specific questions?

v-bbalaiagar commented 3 years ago

Hi @brettsam, Could you please look into this issue.

nilsgaeq commented 2 years ago

Any progress on this issue?

Azure / azure-webjobs-sdk