search

jaegertracing / jaeger

CNCF Jaeger, a Distributed Tracing Platform
https://www.jaegertracing.io/
Apache License 2.0
20.22k stars 2.41k forks source link

Investigate missing operation from jaeger-query metrics reader #3392

Closed albertteoh closed 2 years ago

albertteoh commented 2 years ago

Describe the bug

Context: https://github.com/jaegertracing/jaeger-ui/pull/815#issuecomment-968717141

To Reproduce Steps to reproduce the behavior:

  1. cd docker-compose/monitor/
  2. docker-compose up
  3. /api/metrics/latencies?service=driver&endTs=1636969517360&groupByOperation=true&lookback=7200000&quantile=0.95&ratePer=3600000&step=60000
  4. (Need to confirm) only receiving metrics for operation /FindNearest.
  5. From main Jaeger UI Search tab, select driver service. Both /FindNearest and HTTP GET operations are present.

Expected behavior Receive latency metrics for two operations:

Version (please complete the following information):

Ashmita152 commented 2 years ago

The prometheus query executed while hitting /api/metrics/latencies has label filter as span_kind =~ "SPAN_KIND_SERVER" while the operation HTTP GET has label span_kind = "SPAN_KIND_CLIENT". Hence it gets filtered out.

albertteoh commented 2 years ago

Ah, thanks for investigating, @Ashmita152!

Yes, it's all coming back to me now; we filter on SPAN_KIND_SERVER because we want to aggregate on ingress rather than egress requests (the latter wouldn't make sense IMO).

I'll close this ticket since it's expected (although potentially confusing) behaviour. I'll include this note in the documentation task for ATM.