search

open-telemetry / opentelemetry-dotnet

The OpenTelemetry .NET Client
https://opentelemetry.io
Apache License 2.0
3.09k stars 736 forks source link

Problems with Prometheus on OTel 1.9 #5734

Open dominikjeske opened 5 days ago

dominikjeske commented 5 days ago

Package

OpenTelemetry

Package Version

Package Name Version
OpenTelemetry.Api 1.9.0
OpenTelemetry 1.9.0
OpenTelemetry.Exporter.OpenTelemetryProtocol 1.9.0
OpenTelemetry.Exporter.Prometheus.AspNetCore 1.9.0-beta.2
OpenTelemetry.Exporter.Prometheus.HttpListener 1.9.0-beta.2

Runtime Version

net8.0

Description

Today I tried to use OTel 1.9 but still I have problems with metrics. Prometheus logs:

prometheus[603702]: ts=2024-06-26T13:18:31.847Z caller=scrape.go:1607 level=warn component="scrape manager" scrape_pool=webapi target=
http://xxxxxxxxx:8080/metrics
msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=1

When rollback to 1.7 everything looks good. Same problem was on 1.8 https://github.com/open-telemetry/opentelemetry-dotnet/issues/5613 and issue was closed as resolved.

Is this a bug or we have to use new prometheus to handle metrics from 1.8+?

Steps to Reproduce

Just use standard OTel and serve prometheus endpoint

Expected Result

Prometheus agent is grabbing data with success

Actual Result

Errors from description

Additional Context

No response

Kielek commented 3 days ago

@dominikjeske, could you please provide Minimal, Reproducible Example? If you provide code which we can easily reproduce it should reduce time to solve.

If you find the root cause of the issue/find the way how to solve it will be great if you can share. PR is more than welcomed.

BTW Prometheus is supporting OTLP now. This might be an option to you to solve it (technically workaround, but it is the recommended protocol).