open-telemetry / opentelemetry-dotnet

The OpenTelemetry .NET Client
https://opentelemetry.io
Apache License 2.0
3.18k stars 753 forks source link

Poor Histogram Support for the Prometheus Exporter #5782

Closed SeWieland closed 1 month ago

SeWieland commented 1 month ago

Package

OpenTelemetry.Exporter.Prometheus.AspNetCore

Package Version

Package Name Version
OpenTelemetry.Exporter.Prometheus.AspNetCore 1.8.0-rc.1
OpenTelemetry.Extensions.Hosting 1.9.0

Runtime Version

net6.0

Description

The first issue I experienced: enabling exponential histograms using the Base2ExponentialHistogramConfiguration resulted in no histograms at all. I could not find anything in the docs, until is read #4337

Prometheus exporter: Currently, the exporter ignores exponential histograms.

It would be great to mention this more prominently somwhere in the docs. I then tried to modify the Boundaries of the ExplicitBucketHistogramConfiguration explicitly, because I need much higher values than 10000 - still the default buckets are used.

Is this simply not working, or am I missing something?

I will now try to workaround the whole issue, by pushing metrics directly via the new OTEL endpoint of prometheus instead. If this is the recommended way because many features simply do not work with the prom exporter, please consider leaving a note in the docs as well.

Steps to Reproduce

builder.AddView(instrument =>
{
    return instrument.GetType().GetGenericTypeDefinition() == typeof(Histogram<>)
    ? new ExplicitBucketHistogramConfiguration(){ Boundaries = customBuckets }
    : null;
});

Expected Result

histogram recordings are available in custom buckets in prometheus

Actual Result

histogram recordings are available in default buckets in prometheus

Additional Context

No response

cijothomas commented 1 month ago
image

There is this warning in the doc already about this is under-development. (And no ETA either for making it production ready)

cijothomas commented 1 month ago

Expected Result histogram recordings are available in custom buckets in prometheus

This is not expected. While prometheus exporter does not support Exponential, it does support explicit one, and the custom boundaries should have worked. Would you mind sharing a repro, so it can be looked at?

SeWieland commented 1 month ago

Sorry opening the issue here without investigating it entirely. I think I just got a bit frustrated with the not working Base2ExponentialBucketHistogramConfiguration.. By creating the minimum working example repo for debugging, I found out that it worked as expected. Right now, I have no clue where the issue comes from in our code and will probably request to reopen the issue if it's related at all to the exporter or other otel libraries. But I think its just a misconfiguration after all. Thanks for the heads up anyways