LogCustomMetric with custom dimensions broken for Application Insights ?

[fgheysels]

Describe the bug

I'm using Arcus.Observability to log custom metrics to application insights.
I can see that the custom metric is logged in the underlying log-analytics workspace, and I can also display these custom metrics in a dashboard. However, if a pass in a context dictionary with some custom dimensions for that metric, I cannot split on those dimensions in the dashboard.

I believe this happens because the custom metric is not correctly written to Log Analytics anymore.

When I call the LogCustomMetric method like this:

 _logger.LogCustomMetric("IETS028Processed", 1, new Dictionary<string, object> { ["success"] = successfullyProcessed });

I would assume that I could apply splitting on a dimension called success in the Azure dashboard. However, this is not possible. When I take a look at how the custom metric is written in LogAnalytics, I would think that this is not correctly written.

I can find the metric back in the AppMetrics table, and I would assume that the Properties column in that table contains a simple JSON object with the possible dimensions. The custom metric is available in the AppMetrics, but the Properties colunm contains a bunch of data. This is for instance how it looks (I've copy pasted it and removed some stuff for privacy and because it would be too much clutter:

{"MachineName":"ca-name-7796-mcfbp",{@Metric}",
"Scope":"[\"Job: 7fad7eb3-e5d4-4827-81f4-1ed6edebe687\"]",
"TelemetryType":"Metrics",
"JobId":"7fad7eb3-e5d4-4827-81f4-1ed6edebe687",
"OperationParentId":"34c28949ca8996aa",
"SourceContext":"IETS028MessageHandler","CycleId":"e5595145-77b0-4152-bd28-746577953f49","version":"1.0.0","Metric":"{\"MetricName\":\"IETS028Processed\",\"MetricValue\":1,\"Timestamp\":\"2024-04-26T20:53:57.4507207 +00:00\",\"Context\":{\"success\":true,\"TelemetryType\":\"Metrics\"},\"$type\":\"MetricLogEntry\"}","OperationId":"bb538727ec32d478","ComponentName":"ca-componentname","success":"True"}

Isn't it so that only the contents of the Context property must be written inside the Properties name of the custom metric ? I can see that for the heartbeatstate metric for instance the Context property just a simple key-value pair is.

Additional context Using Arcus.Observability 3.0.0 and Serilog 2.12.0 I'm running my application in a Container App Environment.

[stijnmoreels]

Isn't it so that only the contents of the Context property must be written inside the Properties name of the custom metric ?

No that was not something that was initially the case. The user-provided custom metrics are 'additional' custom metrics. Enrichers are also being part of this, for example. I do not believe that this is a bug; I'm also having trouble seeing what the problem is here. The custom dimension is tracked, right?

[fgheysels]

The custom dimension is tracked, but it's not queryable in Application Insights. I mean, you can display the custom metric in Application Insights, but you cannot 'split' it. Normally, when you have custom dimensions on a metric, you can split visualization of the metric by those custom dimensions. However, the dashboard in Application Insights doesn't allow me to split on it (the button remains disabled). I think this is because the custom dimensions are not correctly written to log analytics.

When I visualize the custom metric in Application Insights, the 'Apply splitting' button is disabled:

I believe this happens because the custom dimensions are not correctly written to the backing table in Log Analytics. This is how they're written to Log Analytics:

As you can see, the 'Properties' column contains quite some information and is actually a JSON object with multiple properties / levels. This JSON actually contains all properties that are normally 'split' and assigned to relevant columns. For instance, I can see that the MetricName, the MetricValue, the Timestamp, etc... are all part of that JSON:

I assume that this JSON is way to complex for Application Insights to parse and perform splitting, and I assume that the JSON in that column should just contain a simple JSON object whch just represents a collection of key/value pairs. That JSON would then just contain the custom dimensions.
They're actually also present in the JSON Object that is now stored in the 'Properties' column. They're present in a property called 'Context':

When looking at other metrics (not logged by us via Arcus), you can indeed see that this Properties column contains a simple JSON object:

When I want to visualize the HeartbeatState metric, I can apply splitting, and you can see that the possible values where you can split on, match with what is present in the 'Properties' column:

[fgheysels]

It reminds me that I have actually exactly the same code for another project where I also log custom dimensions, and there I can indeed split on my custom dimensions:

There, we're using Acus.Observability (Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights) 2.6.0 and have also dependencies on Serilog.Extensions.Hosting 5.0.1, Serilog.Settings.Configuration 3.4.0 and Serilog.Sinks.Console 4.1.0

.edit: when looking at the backing AppMetrics table in Log Analytics, I can see that the Properties column also contains a rather complex JSON structure; cfr with what I have in the project where splitting doesn't work: There goes my theory / assumption.

Did some googling and stumbled upon this: https://github.com/microsoft/ApplicationInsights-dotnet/issues/1144#issuecomment-497362528

I'll have a closer look at that setting which is apparently still applicable.

[fgheysels]

Well, I'll be damned:

So, this is indeed not a bug in Arcus, but a (preview!) setting that must be enabled on Application Insights. A preview setting that exists for more then 5 years!