microsoft / ApplicationInsights-dotnet

ApplicationInsights-dotnet
MIT License
565 stars 287 forks source link

Disable sampling for a specific request #2856

Closed AndyMcKenna closed 8 months ago

AndyMcKenna commented 8 months ago

Is your feature request related to a problem? This isn't a feature request as much as a question on how to do this with the existing features

Describe the solution you'd like. I'd like to be able to entirely disable sampling for a request and anything under that operation (dependencies, traces, events, etc)

Describe alternatives you've considered. I've tried this with a combination of a telemetry initializer and a processor. They are both executing but the requests still get sampled.

Initializer

/// <summary>
///   Factory Method that is used to Initialize by Application Insights
///   If "disableSampling=on" is in the query string, all telemetry for this request will be logged at 100%
/// </summary>
/// <param name="telemetry"></param>
public void Initialize(ITelemetry telemetry)
{
  if (!(telemetry is ISupportProperties telemetryWithProperties) || _httpContextAccessor?.HttpContext?.Request?.Query == null)
  {
    return;
  }

  var disableSampling = _httpContextAccessor.HttpContext.Request.Query.ContainsKey
    ("disableSampling") && _httpContextAccessor.HttpContext.Request.Query["disableSampling"] == "on"
        ? "1"
        : "0";
  telemetryWithProperties.Properties[nameof(GlobalPropertyNamesEnum.DisableSampling)] = disableSampling;
}

and then my processor that inherits from AdaptiveSamplingTelemetryProcessor

/// <summary>
///   If the DisableSampling custom dimension is set, don't subject this item to any sampling checks and log it at 100%
/// </summary>
internal class DisableAdaptiveSamplingTelemetryProcessor : AdaptiveSamplingTelemetryProcessor
{
  private ITelemetryProcessor _next { get; set; }

  public DisableAdaptiveSamplingTelemetryProcessor(ITelemetryProcessor next) : base(next)
  {
    Next = next;
  }

  public DisableAdaptiveSamplingTelemetryProcessor
    (SamplingPercentageEstimatorSettings settings, AdaptiveSamplingPercentageEvaluatedCallback callback, ITelemetryProcessor next) : base
    (settings, callback, next)
  {
    _next = next;
  }

  public new void Process(ITelemetry item)
  {
    if (item is ISupportProperties telemetryWithProperties && telemetryWithProperties.Properties[nameof(GlobalPropertyNamesEnum.DisableSampling)] == "1")
    {
      _next.Process(item);
    }
    else
    {
      base.Process(item);
    }
  }
}

I see all this logic working locally via breakpoints and the results in the portal but in production we sample around 1% and sometimes the dimension will show up but everything under that request still gets sampled out.

I've verified that my processor pipeline is unrelated processors and then DisableAdaptiveSamplingTelemetryProcessor at the end.

Additional context. We get millions of requests per day and have to sample very aggressively to control costs but sometimes I want to be able to capture a totally unsampled request.

cijothomas commented 8 months ago

https://learn.microsoft.com/en-us/azure/azure-monitor/app/sampling-classic-api#frequently-asked-questions check the last qn in this FAQ "There are certain rare events I always want to see. How can I get them past the sampling module?"

AndyMcKenna commented 8 months ago

Thanks!