open-telemetry / prometheus-interoperability-spec

Workgroup for building Prometheus-OTLP interoperability for the OTEL Collector and Prometheus related discussions.
Apache License 2.0
41 stars 6 forks source link

PrometheusReceiver Ignores Timeseries (Histogram and Summary) Metrics without "_sum" counter #26

Closed sankala-dremio closed 3 years ago

sankala-dremio commented 3 years ago

Currently, when Timeseries data is scraped by Prometheus Scraper, it expects bucketed data and two counters along with those buckets - _count and _sum. Some frameworks do no capture the _sum and hence do not produce the _sum counter in the Prometheus Exp format. While the Scraper and Prometheus work fine and display appropriate graphs without this counter, Prometheus receiver on the other hand expects this counter to be present for all time series metrics and if not present silently ignores the metric and associated datapoints.

Is this really the desired behavior for PrometheusReceiver?

jmacd commented 3 years ago

What does the real Prometheus server do in this case? We should do what it does, I think.

sankala-dremio commented 3 years ago

Prometheus Server accepts and renders time series metric irrespective of presence of "_sum". Hence prometheus receiver should at least do the same. I can submit a PR for this change if agreed upon this behavior.

rakyll commented 3 years ago

@odeke-em is working on this.

jmacd commented 3 years ago

Brian B. stated in today's meeting that Prometheus will accept Summaries without "_cnt" as well as without "_sum" and/or without any quantiles.

odeke-em commented 3 years ago

The Prometheus receiver wasn’t necessarily the problem but it was the exporter as I showed in https://github.com/open-telemetry/opentelemetry-collector/issues/2661#issuecomment-813871810 and fixed by PR https://github.com/open-telemetry/opentelemetry-collector/pull/2900

Let’s close this issue as the change to the collector just got merged. Thanks everyone for the discussion.

bogdandrutu commented 3 years ago

@odeke-em it was also this https://github.com/open-telemetry/opentelemetry-collector/pull/2812 that helped

odeke-em commented 3 years ago

Gotcha, thank you @bogdandrutu for the follow up.