Open LGouellec opened 2 days ago
For tracking : https://github.com/LGouellec/streamiz/issues/361
meter?.Dispose(); meter = new Meter("Test");
This looks like a new Meter of same name is created right after disposing it? Not sure what is the intend here.
@cijothomas
meter?.Dispose(); meter = new Meter("Test");
It's just a reproducer, my initial use case is to expose OpenTelemetry metrics for a Kafka Streams application. Don't know if you are familiar with Kafka, but the assignment of partitions can change, so it means you have to drop some metrics in some cases.
Today, we can't drop a metric for a specific Meter instance. Open Issue here : https://github.com/dotnet/runtime/issues/83822 Discussion here : https://github.com/open-telemetry/opentelemetry-specification/issues/3062
So that's why for now, I dispose the current Meter instance, and recreate from scratch for each iteration. With that, I don't need to drop metrics, just not exposing the old metrics in the next iteration.
My point here is when the Meter
instance is disposed, the open telemetry metrics are not released properly.
@cijothomas
Is there a way in OpenTelemetry to clear and dispose all the resources without closing the MeterProvider
?
Hey @LGouellec!
I spent some time on this tonight. Here is what I think is going on...
When you dispose your Meter
we track the Metric
as deactivated: https://github.com/open-telemetry/opentelemetry-dotnet/blob/9f41eadf03f3dcc5e76c686b61fb39849f046312/src/OpenTelemetry/Metrics/Reader/MetricReaderExt.cs#L29 But nothing happens until...
...a collection is run. Once we have exported the data, we clean up the Metric
: https://github.com/open-telemetry/opentelemetry-dotnet/blob/9f41eadf03f3dcc5e76c686b61fb39849f046312/src/OpenTelemetry/Metrics/Reader/MetricReaderExt.cs#L275
The problem is in your repro there you are using AddPrometheusHttpListener
which is a pull exporter. No collection will run unless you cause one.
Do me a favor and switch your code to OtlpExporter
and re-run the test. That is a push exporter meaning it will run periodically and cause a cleanup. I suspect if you run that the memory will be more stable.
Should we do something here? I'm not sure what we could do. Open to ideas. But I will say, what you are proving by constantly disposing Meter
s I don't think is how any of the APIs were designed to be used 😄
Package
OpenTelemetry
Package Version
Runtime Version
net8.0
Description
I need to
Dispose(..)
often myMeter
instance, because today Meter doesn't provide a way to delete an existing metric dynamically. See : https://github.com/dotnet/runtime/issues/83822.When I hit
meter.Dispose()
, the dotnet runtime will notify each instrument perviously created that they are not longer published. See: https://github.com/dotnet/runtime/blob/9e59acb298c20658788567e0c6f0793fe97d37f6/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/Meter.cs#L519The OpenTelemetry
MetricReader
(in my caseBaseExportingMetricReader
) must deactivateMetric : See: https://github.com/open-telemetry/opentelemetry-dotnet/blob/5dff99f8a0b26ff75248d5f2e478b9c3c42f5678/src/OpenTelemetry/Metrics/Reader/MetricReaderExt.cs#L30But it seems there is a memory leak, because when I run my program for a long time, the memory still growing. When I debug my app, I see a lot of MetricPoint[] instances unreleased. More time my application run, more memory consumption is required only for MetricPoint[] instances. It seems when
meter.Dispose()
is called, the open telemetry metrics are not released properly.Screen of my debugger at 10 min uptimes :
Screen of my debugger at 60 min uptimes (122 Mb of MetricsPoint[] )
Steps to Reproduce
Expected Result
All perviously MetricPoint released
Actual Result
Memory leak
Additional Context
No response