Does the SDK not honor Adaptive sampling configuration?

[vany0114]

Hey guys, I'm using the SDK in an Azure Functions environment to customize the way the telemetry is done, especially to correlate requests that the SDK does not support out of the box, such as Event Grid, etc, however, I noticed that it seems it is not honoring the Adaptive sampling configuration, the configuration in my host.json looks like this:

"applicationInsights": {
    "samplingSettings": {
       "minSamplingPercentage": 50,
       "isEnabled": true,
       "excludedTypes": "Event;Request"
    }
}

So I'd expect it would sample the 50% of the dependency calls, traces, and everything that's not in the excludedTypes, however, it's not doing so.

I've also noticed that the SDK exposes a property called samplingPercentage that can be used for kinda the same purposes, however, I guess that's a fixed percentage, isn't it? so for example, if I set it as 50 it would sample the 50% of all kinds of requests, right?

appInsights.defaultClient.config.samplingPercentage = 50

Is there a way to make it work with the Adaptive sampling configuration? or is there a way to instruct it to sample a % of a given type of request, say, that samples 50% of dependencies and 60% of traces?

[vany0114]

cc @JacksonWeber

[hectorhdzg]

@vany0114 unfortunately Node.js SDK does not currently support it, you can find more info about which SDKs support it here

[vany0114]

@vany0114 unfortunately Node.js SDK does not currently support it, you can find more info about which SDKs support it here

@hectorhdzg thanks for the clarification, I guess I can implement it via telemetry processor?

[hectorhdzg]

Telemetry processors are no longer supported in v3 of this SDK, but you should be able to do that using v2, in latest version we are using OpenTelemetry internally, and SpanProcessors or LogProcessors are not designed for sampling, so telemetry cannot be filtered out there, a custom sampler could be used, but it only apply to request/dependency data, logs samplers are still under discussion as far as I know.

[vany0114]

Is there any examples of how to achieve that in v3 using a customer sampler as you pointed out?

[hectorhdzg]

More info about OpenTelemetry samplers here, the actual JS code is available here

[vany0114]

@hectorhdzg quick question, is there a way to configure the log level to avoid ingesting these runtime traces?

Executing 'Functions.HttpTrigger1' ...
Executed 'Functions.HttpTrigger1' (Succeeded ...)

I noticed that using the Application Insights SDK there's no way we can sample those, they never land in the telemetry processor, I guess it's because they are runtime traces.

I know we can configure the log level per function, like:

Function.HttpTrigger1": "None"

However, that's not ideal or maintainable and if I do this:

Function.": "None"

Then I lose all the logs.

We need to do so because they swamped our telemetry data, they are not useful for us and we're being charged for that which is super expensive.

[hectorhdzg]

@vany0114 can you check the SDK version of the unexpected telemetry in Kusto? all telemetry should reach the telemetry processor if using v2.x, if is actually generated by Node.js SDK is possible this is coming from console auto capturing, you should be able to turn that off if that is the case.

[vany0114]

@hectorhdzg this is the query I'm using however I don't see the SDK version there? 🤔 can you please tell me how?

AppTraces
| where TimeGenerated > ago(1d)
| where _IsBillable == true

On the other hand, FWIW the package version we're using is 2.9.1

[hectorhdzg]

Yeah, I don't believe that telemetry is coming from this SDK, @ejizba do you know where is this data being generated and how to configure it?

[vany0114]

FWIW as mentioned before, we're using Azure Functions, so I guess that telemetry comes from the Azure Functions runtime, I also see that all the traces we leave using Context.log never reach the telemetry processors either.

[ejizba]

That telemetry is coming from the Azure Functions host, nothing to do with Node.js.

A few ideas:

1. Do you have to set "Function" or "Function.HttpTrigger1" to "None" or is there a different level like "Error" or "Warning" that would work? I know you said you don't want "Executing" and "Executed" logs, but I'm curious which logs you do want.
2. Sampling settings in host.json should work for these logs. The settings can be confusing (docs here), but if you mess around with them long enough you may get something that works 😅
   • Maybe try removing "excludedTypes". I'm not sure which type the "Executing" and "Executed" logs fall under, but if they match "Event" or "Request" then sampling settings won't apply to them.
   • I think minSamplingPercentage of 50 means the host is allowed to sample between 50 and 100 percent of requests, meaning it'll keep 50 to 100 percent of requests. So if you want to keep at most 50 percent of requests I think you want maxSamplingPercentage instead. You could also use InitialSamplingPercentage set to 50, because I'm not sure what happens if initial is greater than max

Fyi, I'm on the Azure Functions team focused on Node.js. I have some knowledge of the host log settings, but if you want the true experts you probably need an issue in this repo instead: https://github.com/Azure/azure-functions-host/issues

[vany0114]

Do you have to set "Function" or "Function.HttpTrigger1" to "None" or is there a different level like "Error" or "Warning" that would work? I know you said you don't want "Executing" and "Executed" logs, but I'm curious which logs you do want.

I want only the logs we produce, and as I said before, the "Function.HttpTrigger1" configuration is not maintainable, we have a lot of functions + that's easy to miss when a developer adds a new one. The "Function" configuration does not work either as it disregards all the logs even ours, so I cannot set it as "Warning" for example, because then we lose all of our traces.

Sampling settings in host.json should work for these logs. The settings can be confusing (docs here), but if you mess around with them long enough you may get something that works

Trust me, we've tried everything and none of that works, that's why I opened this issue and @hectorhdzg confirmed that that configuration does not work with this SDK.