jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.18k stars 2.4k forks source link

[v2] Use health check extension in e2e test instead of /metrics #5859

Open yurishkuro opened 3 weeks ago

yurishkuro commented 3 weeks ago

In the current e2e tests we are using the metrics endpoint to check that the v2 binary is up and ready for tests:

   e2e_integration.go:98: Checking if Jaeger-v2 is available on http://localhost:8888/metrics

Since we already introduced a health check extension (#5831), we should be using that instead of /metrics.

Changes required:

yurishkuro commented 3 weeks ago

Current state: in #5861 all configs were added, but grpc-storage test failed because additional traces from WriteSpan endpoint are being generated (a big no-no). I am going to merge #5861 with a temporary override in grpc_test to continue using the original query service port for health check, but we need to identify the problem and remove the override for healthcheck endpoint. Surprisingly just switching to query port fixes the issue, while querying 13133 results in extra spans. My hypothesis is that the healthcheck endpoint is registered with tracing enabled, so when we hit if from the test it generates a trace from within the collector, and writing that trace generates the other traces for WriteSpan endpoint that we're seeing. We need to make sure that we do not let OTEL framework instantiate a tracer that indiscriminately traces everything. cc @Wise-Wizard

yurishkuro commented 3 weeks ago

Reproducer in https://github.com/jaegertracing/jaeger/pull/5861#issuecomment-2306270761