Open jakob-o opened 4 years ago
Based on @jack-berg examples, and the possibility to embed extension directly into the agent, I've created this simple project that provides both the java agent and the docker auto instrumentation that will ignore
/health
and/metrics
calls.
life savier, for me this is the easiest solution, but I only needed the jar file:
stages:
- get_jar
- distribute
variables:
OTEL_LIB_VERSION: 1.32.0
Get Jar:
stage: get_jar
image: ghcr.io/vmaleze/opentelemetry-java-ignore-spans:$OTEL_LIB_VERSION
script:
- cp /javaagent.jar ./
artifacts:
paths:
- './javaagent.jar'
expire_in: 1 week
Deploy JAR:
image: maven:3.6-openjdk-17-slim
stage: distribute
rules:
script:
- mvn deploy -DskipTests
I'm one who also waiting for that feature, but I solved this problem from another side (this approach is probably not for everyone).
The task was to disable actuator healthckecks /actuator/health
on tracing UI. So here is what I did.
I've added the filter on opentelemetry-collector's side adding processor filter
to force it to skip spans from actuator.
Here is the documentation about processors: https://opentelemetry.io/docs/collector/configuration/#processors Here is the list of available filters: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor
In total now I have such configuration on my project (which I'll certainly would like to change on config property when it'll be implemented):
config:
exporters:
...
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, filter/drophttp] # do not forget to add processor otherwise it will not work
exporters: [otlp]
processors:
filter/drophttp:
error_mode: ignore
traces:
span:
- attributes["http.url"] == "/actuator/health"
- attributes["url.path"] == "/actuator/health"
I would be appreciated to myself if I would find this comment hours before. 😄
I did the same but I ended up with orphaned spans. How are you dealing with those?
my architecture - app -> otel agent (auto instrumentation) -> otel collector -> observability backend for now, i wanted to filter out swagger and health check calls (caused due to k8s readiness probe)
solution 1 - filter processor. issue - resulted in orphaned spans solution 2 - tail sampling processor. till now, looks like it works
@Kortex, does tail sampling not work?
@shameekagarwal I did not try out tail sampling.
Would it be possible to share your configuration?
i linked my comment, where the code is in a collapsible....expand solution 2
So I decided to abandon the filter/drophttp
processor and after the hours of debugging I've finally end up with the following configuration (thanks to @shameekagarwal):
processors:
tail_sampling:
policies: [
{
name: filter_http_url,
type: string_attribute,
string_attribute: {
key: http.url,
values: [ /actuator/health, /swagger-ui.*, /v3/api-docs.*, /favicon.ico ],
enabled_regex_matching: true,
invert_match: true
}
},
{
name: filter_url_path,
type: string_attribute,
string_attribute: {
key: url.path,
values: [ /actuator/health, /swagger-ui.*, /v3/api-docs.*, /favicon.ico ],
enabled_regex_matching: true,
invert_match: true
}
}
]
NOTE: there are two of them because in your project can be enabled both spring-web
and spring-webmvc
tracing and they are using two different keys containing URL http.url
and http.path
respectively. (personally I decided to disable spring-webmvc
tracing)
otel:
instrumentation:
spring-web:
enabled: true
spring-webmvc:
enabled: true
Not sure why @shameekagarwal has key http.route
, but in my case it is not (Spring Boot 3.2.2; Opentelemetry 1.35.0).
Anyway the good thing that everything is working now. :)
one comment here, tail sampling works absolutely fantastic for almost all issues here (related to actuator) but I have a SSE stream that can be open for more than 1 hour. Tail sampling can't keep up with such long trace.
Is your feature request related to a problem? Please describe. As already mentioned here https://github.com/open-telemetry/opentelemetry-specification/issues/173 I'd like to be able to exclude or sample a list of URLs / URL-Patterns from instrumentation. In my case particularly to avoid generating many events from health- and liveness-checks.
Describe the solution you'd like I opened the issue https://github.com/open-telemetry/opentelemetry-java/issues/1552 to discuss if / how tracing might be disabled from instrumentation. To my knowledge there currently is no API / SDK method to disable tracing centrally on the context. If I understood correctly a (maybe temporary) solution might be to create a non-recording / invalid span in the HTTP instrumentation, which due to the
ParentOrElse
-Sampler, would lead to ignoring child spans as well, if the request matches a URL pattern. Any hint to where this would be architecturally appropriately implemented is highly appreciated. Maybe in the HttpServerTracer?Describe alternatives you've considered We already attempted to use the
otel.trace.classes.exclude
but only succeeded in completely disabling WebMvc instrumentation./CC @gabrielthunig @spaletta