Open chkp-talron opened 8 months ago
Hello, I'm experiencing an issue with the Jaeger Service Performance Monitor (SPM) dashboard where the latency data consistently shows "No Data" for all services. Upon inspecting the query in my web browser, I found that the metrics for "service_latencies" were empty. Here's the information from the query inspection:
{ "name": "service_latencies", "type": "GAUGE", "help": "0.50th quantile latency, grouped by service", "metrics": [] }
I checked the Jaeger documentation and found that the relevant metric for this dashboard is "latency_bucket," which is deprecated on the OpenTelemetry side.
jaeger version: 1.54 opentelemetry collector: 0.94.0
@chkp-talron @MinaMohammadi Am experiencing the same issue. have you found any fixes for this issue??
@ jaeger team @yurishkuro : Can you please look into this issue if not already.
The documentation has quite detailed troubleshooting steps https://www.jaegertracing.io/docs/latest/spm/#troubleshooting. Which of them did you try and what were the results?
The documentation has quite detailed troubleshooting steps https://www.jaegertracing.io/docs/latest/spm/#troubleshooting. Which of them did you try and what were the results?
@yurishkuro Yes, we have indeed followed the well written troubleshooting steps. To be precise we followed the ENV vars mentioned here: https://www.jaegertracing.io/docs/1.57/spm/#viewing-logs PROMETHEUS_QUERY_NORMALIZE_CALLS=true PROMETHEUS_QUERY_NORMALIZE_DURATION=true
Initially we did not see any metrics on the SPM. But after the above inclusion of ENV, the error rate and request rate started to appear, while the latency metrics were not appearing.
Please do let us know if you require any other information, will be glad if you resolve our issue here.
@yurishkuro any further help on this? because of this issue we had to rollback to older opentelemetry version
What happened?
Installed jaeger deployment on k8s cluster, jaeger works properly. I was able to enable the monitor tab by setting various flags in jaeger deployment yamls, but I'm able to see only Request rate data for all of my services , and Error rate data for a single service out of 15 services. but Latency shows "No Data" and the top of the page reads: "No data yet! Please see these instructions on how to set up your span metrics."
i assume that if at least one service show errors data, than the other services does not have errors (the service which shows error if kind of a gateway into our cluster, so it might be reasonable it shows errors and the other dont.)
however, latency is empty all the time for all services. i cant tell if this is due to how metrics are sent to prom, there are tone sof attributes on each on of the metrics, so maybe aggregation needs to be on only a few attributes to be able to show latency data?
I'm sending app metrics using otel SDK, Collector is configured with spanmetrcis connector
Steps to reproduce
Expected behavior
SPM monitor should show latency data
Relevant log output
No response
Screenshot
Additional context
No response
Jaeger backend version
Helm Chart: jaeger-0.73.1 Jaeger version: 1.51.0
SDK
Otel nodejs SDK: "@opentelemetry/api": "^1.4.1", "@opentelemetry/core": "^1.15.2", "@opentelemetry/exporter-metrics-otlp-http": "^0.41.2", "@opentelemetry/exporter-trace-otlp-http": "^0.41.2", "@opentelemetry/sdk-metrics": "^1.15.2", "@opentelemetry/sdk-trace-base": "^1.15.2", "@opentelemetry/sdk-trace-node": "^1.15.2"
Pipeline
OTel SDK -> Otel Collector -> jaeger Collector -> ES
Stogage backend
ES 7
Operating system
Linux
Deployment model
EKS
Deployment configs