Exclude URLs from Tracing

[jakob-o]

Is your feature request related to a problem? Please describe. As already mentioned here https://github.com/open-telemetry/opentelemetry-specification/issues/173 I'd like to be able to exclude or sample a list of URLs / URL-Patterns from instrumentation. In my case particularly to avoid generating many events from health- and liveness-checks.

Describe the solution you'd like I opened the issue https://github.com/open-telemetry/opentelemetry-java/issues/1552 to discuss if / how tracing might be disabled from instrumentation. To my knowledge there currently is no API / SDK method to disable tracing centrally on the context. If I understood correctly a (maybe temporary) solution might be to create a non-recording / invalid span in the HTTP instrumentation, which due to the ParentOrElse-Sampler, would lead to ignoring child spans as well, if the request matches a URL pattern. Any hint to where this would be architecturally appropriately implemented is highly appreciated. Maybe in the HttpServerTracer?

Describe alternatives you've considered We already attempted to use the otel.trace.classes.exclude but only succeeded in completely disabling WebMvc instrumentation.

/CC @gabrielthunig @spaletta

[EvertonSA]

Based on @jack-berg examples, and the possibility to embed extension directly into the agent, I've created this simple project that provides both the java agent and the docker auto instrumentation that will ignore /health and /metrics calls.

life savier, for me this is the easiest solution, but I only needed the jar file:

stages:
  - get_jar
  - distribute

variables:
  OTEL_LIB_VERSION: 1.32.0

Get Jar:
  stage: get_jar
  image: ghcr.io/vmaleze/opentelemetry-java-ignore-spans:$OTEL_LIB_VERSION
  script:
    - cp /javaagent.jar ./
  artifacts:
    paths:
      - './javaagent.jar'
    expire_in: 1 week

Deploy JAR:
  image: maven:3.6-openjdk-17-slim
  stage: distribute
  rules:
  script:
    - mvn deploy -DskipTests

[apischan]

I'm one who also waiting for that feature, but I solved this problem from another side (this approach is probably not for everyone). The task was to disable actuator healthckecks /actuator/health on tracing UI. So here is what I did. I've added the filter on opentelemetry-collector's side adding processor filter to force it to skip spans from actuator.

Here is the documentation about processors: https://opentelemetry.io/docs/collector/configuration/#processors Here is the list of available filters: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor

In total now I have such configuration on my project (which I'll certainly would like to change on config property when it'll be implemented):

config:
  exporters:
      ...

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch, filter/drophttp] # do not forget to add processor otherwise it will not work
        exporters: [otlp]

  processors:
    filter/drophttp:
      error_mode: ignore
      traces:
        span:
          - attributes["http.url"] == "/actuator/health"
          - attributes["url.path"] == "/actuator/health"

I would be appreciated to myself if I would find this comment hours before. 😄

[Kortex]

I did the same but I ended up with orphaned spans. How are you dealing with those?

[shameekagarwal]

my architecture - app -> otel agent (auto instrumentation) -> otel collector -> observability backend for now, i wanted to filter out swagger and health check calls (caused due to k8s readiness probe)

solution 1 - filter processor. issue - resulted in orphaned spans solution 2 - tail sampling processor. till now, looks like it works

@Kortex, does tail sampling not work?

[Kortex]

@shameekagarwal I did not try out tail sampling.

Would it be possible to share your configuration?

[shameekagarwal]

i linked my comment, where the code is in a collapsible....expand solution 2

[apischan]

So I decided to abandon the filter/drophttp processor and after the hours of debugging I've finally end up with the following configuration (thanks to @shameekagarwal):

  processors:
    tail_sampling:
      policies: [
        {
          name: filter_http_url,
          type: string_attribute,
          string_attribute: {
            key: http.url,
            values: [ /actuator/health, /swagger-ui.*, /v3/api-docs.*, /favicon.ico ],
            enabled_regex_matching: true,
            invert_match: true
          }
        },
        {
          name: filter_url_path,
          type: string_attribute,
          string_attribute: {
            key: url.path,
            values: [ /actuator/health, /swagger-ui.*, /v3/api-docs.*, /favicon.ico ],
            enabled_regex_matching: true,
            invert_match: true
          }
        }
      ]

NOTE: there are two of them because in your project can be enabled both spring-web and spring-webmvc tracing and they are using two different keys containing URL http.url and http.path respectively. (personally I decided to disable spring-webmvc tracing)

otel:
  instrumentation:
    spring-web:
      enabled: true
    spring-webmvc:
      enabled: true

Not sure why @shameekagarwal has key http.route, but in my case it is not (Spring Boot 3.2.2; Opentelemetry 1.35.0). Anyway the good thing that everything is working now. :)

[EvertonSA]

one comment here, tail sampling works absolutely fantastic for almost all issues here (related to actuator) but I have a SSE stream that can be open for more than 1 hour. Tail sampling can't keep up with such long trace.