search

open-telemetry / opentelemetry-dotnet

The OpenTelemetry .NET Client
https://opentelemetry.io
Apache License 2.0
3.23k stars 765 forks source link

[bug] Expected value after metric, got "#" ("INVALID") while parsing: "Resilience_polly_str#" #5847

Closed sumjas closed 3 weeks ago

sumjas commented 1 month ago

Package

OpenTelemetry.Exporter.Prometheus.AspNetCore

Package Version

Package Name Version
OpenTelemetry.Api 1.8.0
OpenTelemetry 1.8.0
TBD TBD

Runtime Version

net6.0

Description

I'm facing a recurring issue in my deployed environments while scraping metrics using Prometheus. The error message reads: "expected value after metric, got "#" ("INVALID") while parsing: "Resilience_polly_str#"

I'm using Polly library version 8 for resiliency, which internally uses OpenTelemetry for instrumentation. The metrics are being exported using a Prometheus exporter.

Steps to Reproduce

I'm using Polly library version 8 for resiliency, which internally uses OpenTelemetry for instrumentation. The metrics are being exported using a Prometheus exporter.

Expected Result

It should export the metrics without any error.

Actual Result

"expected value after metric, got "#" ("INVALID") while parsing: "Resilience_polly_str#"

Additional Context

I am also going to raise a ticket with the Polly developer community. Please share the suggestions or workaround for this issue if you have any.

sumjas commented 1 month ago

The Prometheus version we are using - v.2.53.0

Here is the error we are seeing.

image
Kielek commented 1 month ago

I think that you missed one important part of the bug. It is Minimal, Reproducible Example. The same link is available in the bug-wizard.

Kielek commented 3 weeks ago

One month old issue without minimal, reproducible, example . Closing.

@sumjas, feel free to reopen when you provide valid steps to reproduce.

otto-gebb commented 3 days ago

We also have encountered this problem. The error was

expected value after metric, got "#" ("INVALID") while parsing: "process_runtime_dot#"

Looks like an intermittent bug, so no reproduction. The worst thing is, Prometheus seems to give up completely on that target when it encounters such a malformed response, and stops polling it for good, although if it tried, it would "see" that the response is no longer corrupted.