[FEATURE REQ] Controlling sample rate for specific activities

[mortenbock]

Library name and version

Azure.Monitor.OpenTelemetry.AspNetCore 1.1.1

Query/Question

Coming from the classic application insights SDK, it was possible to use a telemetry initializer to set the sample rate of a specific telemetry item, based on whatever your criteria might be.

Is there a way to set a custom sampler when using this package?

builder.Services
    .AddOpenTelemetry()
    .UseAzureMonitor();

Or is there a way to set a different sample rate for a specific activity source?

The use case it that we might have certain requests where we want more accuracy in the data, or we might have a background task making a custom activity, where we always want to include the telemetry from that task.

Environment

Using dotnet 8

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @rajkumar-rangaraj @reyang @TimothyMothra @vishweshbankwar.

[TimothyMothra]

Unfortunately, this isn't supported today. I'll leave this issue open to collect upvotes.

[mortenbock]

@TimothyMothra Would there be a problem with using an approach like this, defining a custom sampler?

builder.Services
    .AddOpenTelemetry()
    .UseAzureMonitor()
    .WithTracing(providerBuilder =>
    {
        providerBuilder.SetSampler(new ParentBasedSampler(new MySampler()));
    });

And then implement a sampler like this:

public class MySampler : Sampler
{
    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        if(samplingParameters.Tags?.Any(t=>t.Key == "AlwaysInclude" && t.Value is true) == true)
            return new SamplingResult(SamplingDecision.RecordAndSample);

        return MakeSamplingDecision();
    }

    private static SamplingResult MakeSamplingDecision()
    {
        // Implement sample rate logic
    }
}

This should allow me to mark a specific activity, and all it's children to always get sampled in, by setting a tag on it?

var tags = new KeyValuePair<string, object?>[]
{
    new("AlwaysInclude", true)
};

using (var rootActivity = OtelStuff.ActivitySource.StartActivity($"MyActivityName", ActivityKind.Internal, null, tags: tags))
{
    // Do some work here...
}

[TimothyMothra]

Hi @mortenbock, Your approach would overwrite our ApplicationInsightsSampler. which we use this to set the ItemCount which is used for some UX experiences.

Alternatively, you could write a custom Filtering Processor See the guide here: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/extending-the-sdk#filtering-processor

The example FilteringProcessor is using OnEnd. Because you called out that you want this to also affect child Activities, you should use override OnStart instead and set:

activity.IsAllDataRequested = true;
activity.ActivityTraceFlags |= ActivityTraceFlags.Recorded;

[mortenbock]

@TimothyMothra Thank you for the input.

I tried with a Filtering Processor, and that does seem to let the activities I want get sampled in.

I did notice however, that if I have the sampling ratio set like this:

    .UseAzureMonitor(options =>
    {
        options.SamplingRatio = 0.2F;
    })

Then my activity gets an itemCount of 5 in the Application Insights data. Even though this particular activity is never excluded from the sampling.

Is there a way to prevent that?

Is there a roadmap for a more flexible sampling approach when using the Open Telemetry packages for Application Insights? It's really the biggest hurdle for us starting the shift from the classic SDK packages currently.

[TimothyMothra]

Hi @mortenbock, sorry for the delay.

1. I tried with a Filtering Processor, and that does seem to let the activities I want get sampled in.

   Can you share your Filtering Processor. We'd like to better understand why this didn't work.

2. if I have the sampling ratio set like this: .UseAzureMonitor(options => { options.SamplingRatio = 0.2F; }) Then my activity gets an itemCount of 5 in the Application Insights data. Even though this particular activity is never excluded from the sampling. Is there a way to prevent that?

   When you set a value of 0.2F, this means that 20% of your traces are sampled, and 80% are dropped. Or to put it another way, 1-in-5 traces are sampled. ItemCount is set to 5 to indicate how many traces that 1 telemetryitem represents. No, there's not a way to prevent that. This is by design.

3. Is there a roadmap for a more flexible sampling approach when using the Open Telemetry packages for Application Insights? It's really the biggest hurdle for us starting the shift from the classic SDK packages currently.

   This is on our backlog, but unfortunately no ETA. We're waiting for the OpenTelemetry Community to define a specification for "adaptive" or "rate-limited" sampling, at which time we will support. cc @mattmccleary

Please continue asking questions, we appreciate your feedback.

[mortenbock]

1. I tried with a Filtering Processor, and that does seem to let the activities I want get sampled in.

   Can you share your Filtering Processor. We'd like to better understand why this didn't work.

Well, the filtering processor did work, but because of the way the ItemCount is calculated, those operation numbers would be way off in the reporting. I've included my processor here:

public class MyFilteringProcessor : BaseProcessor<Activity>
{
    public override void OnStart(Activity data)
    {
        var alwaysInclude = data.GetTagItem("AlwaysInclude") as bool? == true;
        if (alwaysInclude)
        {
            data.IsAllDataRequested = true;
            data.ActivityTraceFlags |= ActivityTraceFlags.Recorded;
        }
    }
}

So the idea here was that the application could tag an activity with "AlwaysInclude", and that would ensure the activity, and all child activities would be sent to Insights.

But without being able to also specify the ItemCount, then this would skew the data.