ITelemetryProcessor does not appear to be supported in Function Apps

[jeffhollan]

From @StevenTaylor on October 5, 2018 5:50

Documentation (https://docs.microsoft.com/en-us/azure/application-insights/app-insights-api-filtering-sampling) suggests that ITelemetryProcessor can be used to filter\sample the telemetry before it is sent App Insights.

But the implementation code is never called in our Function App.

The code to set up the builder is called, which looks like:

var builder = TelemetryConfiguration.Active.TelemetryProcessorChainBuilder; builder.Use((next) => new AppInsightsFilter(next)); builder.Build();

But the 'Process' method implemented in the class is never called, the class looks like:

` class AppInsightsFilter : ITelemetryProcessor { private ITelemetryProcessor Next { get; set; }

    // Link processors to each other in a chain.
    public AppInsightsFilter(ITelemetryProcessor next)
    {
        this.Next = next;
    }

    public void Process(ITelemetry item)
    {
        // To filter out an item, just return
        if (!OKtoSend(item)) { return; }

        // Modify the item if required
        //ModifyItem(item);

        this.Next.Process(item);
    }

    private bool OKtoSend(ITelemetry item)
    {            
        // try something!
        return (DateTime.Now.Second <= 20); // only send for the first 20 seconds of a minute
    }
}`

PS. We have tried setting config values SamplingPercentage (App Insights config) and maxTelemetryItemsPerSecond (host.json) as low as possible to reduce the telemetry data, but there is still too much telemetry.

Copied from original issue: Azure/Azure-Functions#981

[Areson]

@jublair You are correct. I forgot that I was using the 1.0.30-beta1 version.

[Chryogenic]

@Areson - Just to clarify, if using 1.0.30-beta1 version of Microsoft.NET.Sdk.Functions, and setting "EnableDependencyTracking": false E.g.

"applicationInsights": {
      "samplingSettings": {
        "enableDependencyTracking": false
      }
    }

Should filter out DependencyTelemetry?

[brettsam]

Everything under "applicationInsights" maps directly to the ApplicationInsightsLoggerOptions class. "enableDependencyTracking" needs to go under "applicationInsights", not under "samplingSettings".

I've submitted a PR to the docs: https://github.com/MicrosoftDocs/azure-docs/pull/43025.

[Areson]

What @brettsam said. But yes, from my testing it does appear to turn off telemetry tracking in that version.

[grahambunce]

Is this due for fixing any time soon without the hoops that are documented here? These hoops are not working for me anyway in my .net core 2.1 function app using the latest packages (3.0.14). The hack that @marshall76963 doesn't work for me locally in test as the call to this:

var configDescriptor = builder.Services.SingleOrDefault(tc => tc.ServiceType == typeof(TelemetryConfiguration));

returns null. Apparently in my app there are no TelemetryConfiguration classes already registered for this to hook into. I really don't know why :(

I'd rather there was some official answer (and backing documentation) as to how to do this correctly. I'm pretty frustrated that a pretty fundamental thing is so lacking. Even this suggested way of using your own telemetry initializer in Azure Functions

services.AddSingleton<ITelemetryInitializer, CustomTelemetryInitializer>()

in order to change the RoleName doesn't seem to work when there are other GitHub tickets that say it does.

Can we have an estimated fix for a proper way of doing this?

[brettsam]

What version of Microsoft.Net.Sdk.Functions are you referencing? You may be hitting this: https://github.com/Azure/azure-functions-host/issues/5530#issuecomment-582553472.

The fix will be coming soon.

[espray]

@grahambunce ran into the same issue See https://github.com/Azure/azure-functions-host/issues/3741#issuecomment-507264617

[grahambunce]

@brettsam I'm referencing what I believe to be the last available version for .net core 2.1.x, i.e. Microsoft.NET.Sdk.Functions 1.0.31.

[grahambunce]

@brettsam is there a release date? I'm hoping this will also clear up the significant amount of "noise" in AI trace logs that Azure Functions spit out. I can't seem to prevent them appearing in the logs, e.g.

• Whenever a service bus message has been detected
• Whenever a service bus message has been successfully processed
• Trigger details
• Timer triggers firing and completing etc.

I don't care about any of these at TRACE level - all I want are my TRACE logs and i can't find a way to not log these system messages

[OskarKlintrot]

Does it not work to ignore their namespaces in appsettings.json?

[brettsam]

@grahambunce -- you should be able to filter any messages you want by their Category (or even a partial match on their Category) in your host.json. See this for details: https://docs.microsoft.com/en-us/azure/azure-functions/functions-monitoring#configure-categories-and-log-levels.

For example, the host.json below would turn off all logging except:

• The Host.Results category of logs (https://docs.microsoft.com/en-us/azure/azure-functions/functions-monitoring#category-hostresults)
• Any logs that start with MyNamespace would be filtered at the Trace level (in other words, not filtered).

{
  "version": "2.0",
  "logging": {
    "logLevel": {
      "default": "None",
      "MyNamespace": "Trace",
      "Host.Results": "Information"
    }
  }
}

[grahambunce]

@brettsam thanks - that does work. I was confused as I use the AI NLog adapter and was trying to filter it there via NLog but it was making no difference. I guess these host logs use their own logging mechanisms.

[brettsam]

A quick summary of what the flow looks like is (hopefully this helps):

RAW LOGS --> LIVE METRICS --> FILTER --> "PERSISTED" APP INSIGHTS

[Areson]

I think there are two issues outstanding with the filtering for me, though I have hacked around it with the solution I outlined above:

1. I can't filter Live Metrics. As @brettsam Pointed out, the native filtering doesn't take place until after live metrics are processed. The raw stream of data is too verbose to make it useful in the live stream.
2. I don't have the level of control in the existing filtering that I need. There is a lot of noise due to certain behaviors being noted as "errors" that really aren't. An example would be rest API call that result in non-200 responses but aren't really issue, like a 404 not found response that I use to trigger an action. These are expected responses and don't really represent an error or failure in terms of the application but will show up as a failed dependency regardless. The sheer volume of these can hide real issue. Turning of all dependencies is not an option either, because then I wouldn't see the actual errors.

[brettsam]

For number 1 -- I've repurposed an existing issue for tracking and added a proposal for an improvement here: https://github.com/Azure/azure-webjobs-sdk/issues/2447.

For number 2 -- yeah, this is a problem we've seen with the auto-dependency tracking in general. We've changed our code in a few places to reduce 404 and 409s just to address this. Sometimes "error codes" from REST calls aren't really errors, yet App Insights reports them as such by default. One option is to register your own ITelemetryInitializer with DI that can check for your APIs that you want to mark as success and do it there. @cijothomas -- do you know if there's any other way to manage this?

[cijothomas]

The recommended way in Application Insights to deal with these is to write telemetry processor to filter out unwanted dependencies. Unfortunately there is no easy way to hook up processors in Functions. So writing TelemetryInitializers to override fields (success) is best bet until telemetry processor support is enabled.

[stap123]

@brettsam Any development on this, is it likely to be added to a sprint any time soon?

[NateB2]

We're running into the issue described in https://github.com/Azure/azure-functions-host/issues/3741#issuecomment-584902726 issue locally (haven't tried Azure yet) when using Functions V3. We are trying to filter out synthetic traffic (FrontDoor/Traffic Manager probes) using a custom telemetry processor. Currently, since there is no service type of "TelemetryConfiguration" being registered, we can't hook into it and add our own processor. We're using the 3.0.7 SDK version, and I tried using the 2.11.0 version of App Insights, but that didn't fix it.

[metoule]

Note that using the extension method AddApplicationInsightsTelemetryProcessor doesn't work either (from Nuget Microsoft.ApplicationInsights.AspNetCore v2.8.0).

[vitalybibikov]

The same here, TelemetryConfiguration is always null for 3.0.7

[brettsam]

@vitalybibikov, some initial questions:

1. Are you seeing this locally or in production?
2. Do you have APPINSIGHTS_INSTRUMENTATIONKEY set? Even locally -- setting it to some dummy string (like "abc") will cause the TelemetryConfiguration to be registered.

If both of those are yes, can you share a sample application that reproduces this? Even a csproj file may be enough.

[NateB2]

Adding the "APPINSIGHTS_INSTRUMENTATIONKEY" fixed it for me. I only previously tested it locally, didn't test it in Azure.

[vitalybibikov]

I've checked it out, when APPINSIGHTS_INSTRUMENTATIONKEY is set, instance is no longer null.

Is it reflected in docs? if it's not, maybe it should be reflected, as it's not quite obvious.

Thanks.

[kamilzzz]

What's the current state of this issue?

I've tried code posted by @michaeldaw and my custom TelemetryProcessor was indeed called but then Live Metrics in the portal are broken. Now they are showing only CPU/Memory usage. Cannot see traces/dependencies anymore (they are sent for sure because after a few minutes I can see them in the Performance -> Dependencies UI).

[luthus]

Has there been any progress on fixing this issue? I'm still seeing the issue with Azure Functions V3.

[luthus]

This change appears to resolve the issue: https://github.com/luthus/azure-webjobs-sdk/commit/3137c5c8e59fdd2495c4ad0b4a09b8748f7ee1f9

With this change we should be able to use builder.Services.AddApplicationInsightsTelemetryProcessor() in the same way that we do for ASP.NET Core.

[jschieck]

@luthus's solution above is the correct one, but if you don't want to have to fork the web jobs sdk, you can get it working correctly in durable functions (or azure functions in general) WITHOUT breaking live metrics dependency and error logging like @kamilzzz saw and I did as well with @lmolkova's solution.

Adding the ITelemetryModule and inserting your ITelemetryProcessors into the builder there works as expected. This gets around builder.Services.AddApplicationInsightsTelemetryProcessor() not working as expected.

// startup.cs
builder.Services.AddSingleton<ITelemetryModule, MyCustomTelemetryModule>();
builder.Services.AddApplicationInsightsTelemetry(Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY"));

// custom module
public class MyCustomTelemetryModule : ITelemetryModule
{
    public void Initialize(TelemetryConfiguration configuration)
    {
        // add custom processors
        configuration.TelemetryProcessorChainBuilder.Use(next => new MyCustomTelemetryProcessor(next));
        configuration.TelemetryProcessorChainBuilder.Build();
    }
}

// custom processor
public class MyCustomTelemetryProcessor : ITelemetryProcessor
{
    private readonly ITelemetryProcessor _next;

    public MyCustomTelemetryProcessor(ITelemetryProcessor next)
    {
        _next = next;
    }

    public void Process(ITelemetry item)
    {
                bool myCustomSkipTelemetry = false;
        if (myCustomSkipTelemetry)
            return;
        _next.Process(item);
    }
}

[luthus]

@jschieck that works as a workaround for now.

[DarinMacRae]

@jschieck This is good!

I just want to clarify something though.

The docs specifically say not to add AddApplicationInsightsTelemetry...

here: https://docs.microsoft.com/en-us/azure/azure-functions/functions-dotnet-dependency-injection#logging-services

...but in your testing you found that it is ok to do so when customizing the processors?

[jschieck]

@jschieck This is good! I just want to clarify something though. The docs specifically say not to add AddApplicationInsightsTelemetry... here: https://docs.microsoft.com/en-us/azure/azure-functions/functions-dotnet-dependency-injection#logging-services ...but in your testing you found that it is ok to do so when customizing the processors?

The app we're doing this in runs in a kubernetes cluster not default azure functions so I'm not sure if different rules apply. But we are still getting all of the built in ILogger/app insights functionality from the functions host runtime without any side effects (that we've noticed)

Also I haven't tested in a normal function app if builder.Services.AddSingleton<ITelemetryModule, MyCustomTelemetryModule>(); is all you would need to get it working

[gabrielweyer]

@jschieck I tested your code sample in a "normal" Azure Function. I can confirm that the telemetry initializers and telemetry processors are called as expected (whereas telemetry processors are not being called for request telemetry items using the previous solution posted in this thread).

The drawback of your approach is that telemetry processors are being called for dependencies that are internal to the Azure Functions SDK. These dependencies are being discarded beforehand using the approach I linked to above. Example of such internal dependencies are retrieving new messages for the Service Bus trigger or the blob used for locking. In my sample Function App with a single Service Bus trigger, 135 "internal" dependencies were recorded in 60 seconds. These dependencies do end up being discarded before being ingested by Application Insights but that is still a lot of telemetry going through the processors.

[ssa3512]

Would it be possible to get the PR for this (https://github.com/Azure/azure-webjobs-sdk/pull/2657) reviewed and merged so users can start utilizing custom telemetry processors? I am in the process of trying to stand up a new functions app that is registering very large numbers of dependency telemetry so utilizing a custom filter to exclude this is a must have.

[piotrzbyszynski]

So 2 and a half years have passed since original issue was created. As I understand it is still not resolved, just workarounds?

[piotrzbyszynski]

So I have to filter out an exception, a false positive (for details see https://github.com/Azure/azure-webjobs-sdk/issues/2612, the issue is completely ignored for over half a year now), filtering this exception is a workaround all together, since this exception should not be logged at all... But anyways...

I can not use category filtering because exception is thrown by Entity Framework code, the category is Microsoft.EntityFrameworkCore.Update and I want to log this exceptions except exceptions with specific message.

I developed a telemetry processor and added it using custom ITelemetryModule approach, and it seems to work locally. Unfortunately when deployed to Azure exceptions are not being filtered. I use Microsoft.ApplicationInsights.AspNetCore version 2.15 and I can not downgrade it to 2.6.1 as suggested here because I also use EnableSqlCommandTextInstrumentation configuration

@brettsam @lmolkova the issue is not fixed for over 2 years now. This issue makes us loose money on AI fees and even more money on developing and researching workarounds. Could we please have this issue fixed once and for all? Pretty please?

[luthus]

@brettsam @lmolkova can the team take a look at this PR? https://github.com/Azure/azure-webjobs-sdk/pull/2657

[piotrzbyszynski]

One more thing regarding category filtering. I have a function called let's say MyFunction, triggered by service bus message. The default category for this function is Function.MyFunction. To this category belongs tons of noise I do not care about like:

Executing 'MyFunction' (Reason='(null)', Id=ee1c2eb3-73ec-417b-a552-70cbab784d17)

or

Trigger Details: MessageId: XYZ, SequenceNumber: 123, DeliveryCount: 6, EnqueuedTimeUtc: 2021-06-01T11:27:42.2850000Z, LockedUntilUtc: 9999-12-31T22:59:59.9999999Z, SessionId: ASD

But also, inside MyFunction I create custom dependency like so:

using (var operation = _telemetryClient.StartOperation(new DependencyTelemetry()
            {
                Name = "My function",
                Type = "MyFunctionOperation",
            }))

My question is: how can I assign custom category to this dependency, because it has Function.MyFunction category assigned by default, and I can not see a way to change that. That means I can not filter out Function.MyFunction category using host.json. Also all DB and HTTP calls have Function.MyFunction category assigned (and I want it to be logged). Any workaround or preferably legitimate solution other then telemetry processors that do not work @brettsam @lmolkova?

[piotrzbyszynski]

Another month has passed without even single response from MS... 🤦‍♂️

Any of you guys can recommend some good Application Insights replacement?

[gabrielweyer]

I came up with a solution that calls telemetry processors on request telemetry items while re-using the existing processor chain (so that the Functions internal dependencies are discarded before hitting the user's telemetry processors). I was hoping that Azure Functions v4 would support telemetry processors but it does not (at least for the in-process hosting model). v4 emits a bit more telemetry and Azure.Core emits a lot more telemetry.

I understand that using telemetry processors can skew the statistics and that sampling is the preferred approach but sometimes I need to discard a specific telemetry item and I'm happy for sampling to apply to everything else. Without any configuration some Functions will incur an Application Insights bill 10 times bigger than the compute bill!

Adding telemetry processors support to Azure Functions would make it a more compelling offering. Currently we have to take into consideration the additional Application Insights spending.

The solution I came up with has been inspired by the other workarounds posted on this thread. I created a library out of it which hopefully will be easy enough to integrate into any Functions App. The code is a bit too long to post here, the interesting part is the extension method configuring Application Insights.

Using this customisation:

• You can add any numbers of telemetry processors in the same way you would add them in ASP.NET Core
• When APPINSIGHTS_INSTRUMENTATIONKEY / APPLICATIONINSIGHTS_CONNECTION_STRING is not present, I register a no-op TelemetryConfiguration so that it can be resolved by the container (no more exception when Application Insights is not configured)
• Most of the "cruft" telemetry emitted by the runtime is discarded

The project comes with a couple of sample Functions so that you can see how it behaves.

Hopefully this helps someone else, it has helped us reduce our Application Insights spending while retaining the ability to troubleshoot issues.

[mrebuffet]

We are also impacted by this issue and have tried several options to fix this behaviour. Our Functions are sitting behind FrontDoor and we are using the built-in Health check option to probe our services. Since Azure has a significant amount of Point of Presence around the world (which is great), this results in almost 200k logs per day per backend pool on the lowest setting (probing at a 255 second interval).

Using the built-in filtering option, we were only able to hide the traces (Executing 'HealthProbe' / Executed 'Health Probe') but couldn't find any out of the box way to filter out the ~70k requests generated in the process.

@gabrielweyer solution is indeed allowing us to filter out all the unwanted requests but forces us to import a new package and fix our custom implementation or bring in a lot of code that may break with future releases.

Before we spend more time looking into this and implementing an alternative, is there any update on this issue or future plans to allow developers to filter out requests by name? Is this active PR still being looked at? https://github.com/Azure/azure-webjobs-sdk/pull/2657

[gha-zund]

We struggle with very high cost for application insights and the solutions and workaround proposed for this issue would help us to reduce the cost and still be able to use application insights. In my opinion, filtering should be possible with configuration, without need to add code which has a high chance to break with future SDK or Function runtime releases...

The fact that this issue is open for almost four and a half year now will have some weight in a decision for another monitoring solution...

Additionally, I would expect to learn all this things I can read in this thread in the documentation without having to read through a GitHub-issue!

[drilko]

What is the status on this? It will soon be 5 years since this was opened. Please provide some feedback.

[JackD111]

I've encountered the same issue when first using isolated Azure Functions. The host manages the RequestTelemetry sent to Application Insights, and there's no way to control those requests from the worker (without completely disabling request telemetry data). Therefore, I developed a custom worker extension to address this.

The repository is available here and is also published as a NuGet package that can be referenced in your isolated azure function project.

This worker extension introduces custom HTTP trigger function bindings that incorporate a custom ITelemetryProcessor. This processor can:

1. Change the RequestTelemetry.Success property to false if the trigger response yields an unsuccessful status code.
2. Discard successful RequestTelemetry events

Discarding successful request events has been a primary objective, significantly reducing our Azure analytics ingestion costs. These costs were primarily driven by front door health probes, which generated gigabytes of data daily.

This should assist others looking to optimize their Azure analytics expenses.

[pregress]

@JackD111 thanks for saving the day your nuget package is exactly what we needed

[federiconintex]

@JackD111 I am facing the same issue (RequestTelemetries being handled by host only) and had a look at your project. I tried to replicate it to adapt it to my requirements but I had issues with the extension startup not being invoked.

1. Must the extension provide custom bindings to be invoked? Is it possible in your knowledge to have the extension code take over the TelemetryConfiguration without having to define those?
2. Do you have some source material to learn more about taking over host functionalities? I am struggling to find any good docs