[App Insights] Allow for filtering of Live Metrics

[kgoderis]

Recent function hosts generate a lot of

Poll for function '...' on queue '....' with ClientRequestId '55e88d2f-7f61-4897-bb10-6e459a2c0520' found 0 messages in 6 ms.

messages that are visible in the "Live Metrics Stream" on Azure Application Insights, but these do not show up when searching in the "Search" functionality of Application Insights.

In the logged entries there is no indication of the LogLevel or so.

How can these entries be filtered or disabled?

[kgoderis]

When the function runs locally (v2.7.1724) then this telemetry is tagged with category Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener. However, running the same code on Azure, the telemetry is not tagged with this category. On the same page, telemetry in the cloud gets a prop_messageCount property, whereas local telemetry does not.

Adding a line to the "logging":"logLevel" section in host.json, for example,

        "Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener": "Error"

does not change a dime, e.g. this line is seemingly ignored and all telemetry of that category is logged.

[brettsam]

Are you trying to have these persisted in App Insights? Or are you trying to hide them from Live Metrics?

The Live Metrics Stream is meant to be a live "snapshot" into everything happening (and that data is stored in-memory; not all of it is persisted anywhere) -- after we send details to that stream, the filter specified in host.json is applied. There's currently no way to filter from the Live Metrics stream, other than using the Filter built-in to the portal (the little filter next to Sample Telemetry). Using that, you can completely remove traces, or tell it to ignore anything with "QueueListener", for example.

[kgoderis]

Are you trying to have these persisted in App Insights? Or are you trying to hide them from Live Metrics?

The Live Metrics Stream is meant to be a live "snapshot" into everything happening (and that data is stored in-memory; not all of it is persisted anywhere) -- after we send details to that stream, the filter specified in host.json is applied. There's currently no way to filter from the Live Metrics stream, other than using the Filter built-in to the portal (the little filter next to Sample Telemetry). Using that, you can completely remove traces, or tell it to ignore anything with "QueueListener", for example.

Indeed, I try to filter in in the live metrics stream. I understand the solution, but why doe other filter.json filter seemingly work?

[brettsam]

The host.json you mean?

We made a decision a while ago that we wanted to show everything in the Live Metrics stream, regardless of filter. Since these records aren't persisted anywhere and you're not charged for Live Metrics usage, it wasn't considered an issue.

The best example of why we do this -- you may have very high througput and not want to log any RequestTelemetry, instead relying on the Metrics we log with our Aggregator. But we still wanted you to be able to get a concrete view of the current state in Live Metrics. For that reason, we allow everything through to Live Metrics.

So, we "capture" your filter, and apply it after the details are sent to Live Metrics, but before they're sent to be persisted by App Insights.

I understand it's confusing -- we may re-think how we can make this more obvious.

[diegosucaria]

Hello, I'm facing the same issue as well. It fills the stream with garbage and makes it quite unusable.

I know I can filter it, but since the "filters" can't be saved in the live metrics stream, it is really a pain to do it every time.

[AartBluestoke]

is there an appropriate place to raise "we should be able to safe filters on live stream" as an issue?

[brettsam]

We're getting more-and-more folks coming to us with confusion around this and filtering. We need to do something to make this more clear. We could do something like:

1. By default, for Traces only, obey the filters that were applied and allow all other logs to flow.
2. Allow some strict "apply all filters to Live Metrics" option that obeys the filters for everything.
   • This could be triggered by adding a specific "liveMetrics" section to the logging filters. This behavior already exists for other loggers so wouldn't be a completely new concept.

I think that could strike the right balance of being able to use Live Metrics out-of-the-box for most folks, and also allowing them to control things directly if they want.

[biltongza]

Any updates on this?