open-telemetry / opentelemetry-python

OpenTelemetry Python API and SDK
https://opentelemetry.io
Apache License 2.0
1.66k stars 568 forks source link

PeriodicMetricExporter does not export periodically for either Delta or Cumulative temporality ExponentialHistogram #3971

Open alexchowle opened 2 weeks ago

alexchowle commented 2 weeks ago

Describe your environment

OS: Centos 7 Python version: 3.11.1 SDK version: 1.25.0 API version: 1.25.0

What happened?

I have a low-throughput service and have implemented my own Histogram to measure the duration of an operation. This uses a PeriodicMetricExporter which is configured to export every 60 seconds. When I use Delta temporality, I get an export within 60 seconds of the last histogram recording. I then have no exports until the next observation is made.

When changing to Cumulative temporality I have the exact same experience. For comparison purposes, when using the Golang SDK I get the same behaviour as Python's Delta but, when using Cumulative, I get a metronomic export every 60 seconds regardless of any observations being recorded..

Should the Python PeriodicMetricExporter export every 60 seconds or not, and how does temporality setting play into it? At the moment it causes me a problem because my service has low throughput. My observability backend (New Relic) uses the StartTimeUnixNano value which can be hours in the past. A constant periodic export would help to correct this as the StartTimeUnixNano would be at most 60 seconds in the past.

Steps to Reproduce

metrics_temporality = {
        Histogram: AggregationTemporality.DELTA
    }
otlp_metric_exporter = OTLPMetricExporter(endpoint=grpc_collector_address, insecure=True,
                                              preferred_temporality=metrics_temporality, timeout=30_000)
metric_reader = PeriodicExportingMetricReader(otlp_metric_exporter,
                                                  export_interval_millis=60_000))
meter_provider = MeterProvider(metric_readers=[metric_reader], resource=_create_resources(),
                                   views=[View(instrument_type=Histogram,
                                               aggregation=ExponentialBucketHistogramAggregation())])
metrics.set_meter_provider(meter_provider)

meter = metrics.get_meter(__name__)

h = meter.create_histogram(
        name='my_histogram', unit='s', description='just my histogram'
    )

h.record(1.0)
time.sleep(120)
h.record(2.0)

Expected Result

I expected to see periodic metric exports for the histogram every 60 seconds, regardless of chosen temporality

Actual Result

A metric export within 60 seconds of the last observation.

Additional context

N/A

Would you like to implement a fix?

None

aabmass commented 2 weeks ago

@ocelotl