Tail-sampling evaluation completing but not filtering out any traces for Grafana Tempo Distributed / agent

[gmcwhirt]

What's wrong?

No response

Steps to reproduce

I am not entirely sure if this a bug or not, but I am using tail sampling to filter out basic endpoints for Grafana Tempo Distributed.

Steps to reproduce I don't have the exact steps to reproduce this config not to work, but right now my log is saying this:

level=debug component=otelcol.processor.tail_sampling.tempoFilter msg="Sampling policy evaluation completed" batch.len=0 sampled=0 notSampled=0 droppedPriorToEvaluation=0 policyEvaluationErrors=0

My understanding is that since the batch.len is = 0, that for whatever reason my traces are not getting to my batch. I am using the receiver and batch Otel.processor as well and they are set up for the receiver to send the output to the batch and has batch sending enabled..

What did you expect to see? I expect for my logs to show that it is filtering out some of these traces that match the endpoints in the config. I have tried invert match and that did not work either...

What did you see instead? A clear and concise description of what you saw instead.

What version did you use? Kubernetes: 2.21.1

Here is the config that I am using:

otelcol.processor.tail_sampling "tempoFilter" {
  policy {
    name = "generic-endpoint-filtering"
    type = "string_attribute"

    string_attribute {
      key                    = "http.url"
      values                 = ["/health", "/info","/prometheus" ]
      enabled_regex_matching = true
      invert_match           = true
    }
  }

  buffer_duration = 60 seconds

  sampling {
    type = "head"
  }

  policy {
    name = "generic-method-filtering"
    type = "string_attribute"

    string_attribute {
      key                    = "http.method"
      values                 = ["OPTIONS"]
      enabled_regex_matching = true
      invert_match           = true
    }
  }
}

Environment MAC Ventura 13.4

Additional context Add any other context about the problem here.

System information

No response

Software version

No response

Configuration

No response

Logs

No response

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[rfratto]

Hi there :wave:

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)