Open chenlujjj opened 2 months ago
Found a PR: https://github.com/grafana/tempo/pull/3917 may be related to the tempo_receiver_accepted_spans
metric, I'll try to upgrade my tempo deploy
Hi, the tempo_receiver_accepted_spans will be available in the 2.6.0 release and then it will need to be updated in the helm chart. For the tempo_build_info metric, we use the same dashboards so it makes sense for us, maybe @zalegrala knows more.
Thanks @javiermolinar
Does the tempo_build_info
metric in your stack have cluster
label?
Below is what I get from one of the temp distributor instances:
Here is where is populated: https://github.com/grafana/tempo/blob/fbf249a41fdc9ee9ddc8168c4a4f92e426f92bb0/cmd/tempo/build/build.go#L20
The cluster label is probably added in the K8s relabel configuration. That way all our metrics include the cluster info
Got it!
That's right. Add a cluster
and namespace
label in the scrape configs. This should mean the queries in the dashboard work as intended.
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.
Describe the bug
Hi team, I deployed tempo-distributed in k8s cluster and tried to monitor it with the dashboards here. But I found that the dashboards are broken, for example:
cluster
label intempo_build_info
metric, which causes the dashboard variablecluster
no value at alltempo_receiver_accepted_spans
metric from the distributor, but it is used in the distributor monitor panelExpected behavior
The dashboards should be normal and show metrics well
Environment:
Additional Context