Tempo (v2.5.0) Distributor missing tempo_receiver_accepted_spans

[jameshounshell]

Describe the bug Today I upgraded the tempo-distributed helm chart from 1.8.5 to 1.10.0 (app version 2.3.1 to 2.5.0) and I noticed in the Tempo Operational dashboard, that the tempo_receiver_accepted_spans metric disappeared. I'm assuming this the result to a change in the embedded OTLP library but other than the change noted in the release, there was nothing specific about this metric, making me think breaking it was unintentional.

To Reproduce Steps to reproduce the behavior:

1. Install version 1.10.0 of the tempo-distributed helm chart.
2. Verify the absence of tempo_receiver_accepted_spans at /metrics of the tempo distributor component.

Expected behavior I would expect the metric tempo_receiver_accepted_spans to still exist/be-emitted.

Environment:

• Infrastructure: GKE fully managed
• Deployment tool: Kustomize plus helm (helmCharts) chart

Additional Context Here is a graph showing tempo_receiver_ metrics disappearing while tempo_distributor_spans_received_total remains.

[joe-elliott]

Thank you for the issue. This is likely due to an upgrade of OTEL:

https://github.com/grafana/tempo/pull/3515 https://github.com/grafana/tempo/pull/3148

I would recommend using the metric you pointed out tempo_distributor_spans_received_total. I would like to restore this metric, if someone wants to attempt a PR.

[akasarapu]

I'm also facing the same issue, tempo_receiver_accepted_spans and tempo_receiver_refused_spans metrics are missing after I have upgraded my tempo chart to 1.10.0 from 1.9.1 and image to 2.5.0 from 2.4.1.

@joe-elliott/ @jameshounshell, Any idea on if this issue fixed or still exists?. Or any work around we have to do to make it work?

[joe-elliott]

It still exists. We would need to find a way to extract that metric again from the otel receivers to publish it.