open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.89k stars 2.27k forks source link

[processor/tailsamplingprocessor] documentation to drop trace #34145

Open mangkoran opened 1 month ago

mangkoran commented 1 month ago

Component(s)

processor/tailsampling

Describe the issue you're reporting

In tailsampling readme there is no mention on how to drop a trace. What I understand from this part is that each policy will result in either "sample" or "not sample" decision. However, there is also no mention how to create a "not sample" decision.

How can I drop a certain trace? e.g. match the trace using string_attribute and drop the matched trace. Initially I thought this could be done using filter processor. However, as stated in https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/31355 filter processor can only drop span and not the entire trace.

github-actions[bot] commented 1 month ago

Pinging code owners:

jpkrohling commented 1 month ago

You can set an "invert" match: if a trace has a span with a specific attribute value for a given attribute, it would normally yield a "Sample" decision. An inverted match would yield a "Not Sample" decision.

mangkoran commented 1 month ago

Thank you for the explanation. However I still couldn't understand the explanation in the docs:

  • When there's an "inverted not sample" decision, the trace is not sampled;
  • When there's a "sample" decision, the trace is sampled;
  • When there's a "inverted sample" decision and no "not sample" decisions, the trace is sampled;
  • In all other cases, the trace is NOT sampled

There's "inverted sample" mentioned. This is contradictory to your explanation:

An inverted match would yield a "Not Sample" decision.

From how I understand the docs, there could be 4 policy outcome:

  1. Sample, not inverted
  2. Sample, inverted
  3. Not sample, not inverted
  4. Not sample, inverted

CMIIW

jpkrohling commented 1 month ago

I think both are right:

Here's an example from the docs:

         {
            name: test-policy-9,
            type: string_attribute,
            string_attribute: {key: http.url, values: [\/health, \/metrics], enabled_regex_matching: true, invert_match: true}
         },

In this case, if the URL contains /metrics, we are dropping the trace.