search

open-telemetry / opentelemetry-dotnet

The OpenTelemetry .NET Client
https://opentelemetry.io
Apache License 2.0
3.23k stars 764 forks source link

[feature request] Add option to remove metric series that are no longer present in observable measurements #5950

Open Noahnc opened 1 week ago

Noahnc commented 1 week ago

Package

OpenTelemetry

Is your feature request related to a problem?

No response

What is the expected behavior?

We are using the ObservableGauge provided by the dotnet Meter to create dynamic metric series, that are then exported by the OTEL-Exporter and sent via Grafana Alloy to Grafana (Mimir). As far as we can tell, there is currently no way to remove a metric series from the exporter once a measurement was collected. When no measurement is provided for a series by the measurement function, the exporter continues to export the last known measurement value until the application is restarted. We have some use-cases with dynamic data, where we need to be able to stop exporting a series without restarting the entire application.

Which alternative solutions or features have you considered?

We already considered setting the value of the series to a pre-defined value like 0 or -100, to then filter all series with this value in Grafana or an Open Telemetry Processor (like Grafana Alloy). This works for some of our use-cases, but still has a lot of configuration overhead.

Additional context

No response

cijothomas commented 1 week ago

I believe this is the same bug shown here : https://github.com/open-telemetry/opentelemetry-dotnet/pull/5952/files

Noahnc commented 1 week ago

Hello @cijothomas

Thanks for pointing out your pull request. It seems like you are right regarding the SDK not following specs in asynchronous collection. The behavior described here is precisely what we need.

cijothomas commented 1 week ago

@Noahnc Would you have time/interest in checking and offering a PR with a fix?