search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
29.02k stars 3.56k forks source link

Scraper fails on prometheus NaN metrics #23287

Closed andig closed 1 year ago

andig commented 2 years ago

Steps to reproduce:

  1. Add a scrape target the has NaN values
  2. Scrape it

Expected behavior: No errors

Actual behavior: Log file:

ts=2022-04-21T06:05:39.347453Z lvl=error msg="Unable to marshal json" log_id=0_zV5qh0000 error="json: unsupported value: NaN"

At this point, no metric is collected.

It seems this behaviour has recently been introduced with the prometheus golang library in 1.12.1: https://github.com/prometheus/client_golang/issues/981. According to spec (see that issue), NaN is a value.

A previous issue for Inf values was fixed in https://github.com/influxdata/influxdb/issues/10490.

Environment info:

InfluxDB 2.0.9 Server: d1233b7 Frontend: 9c89e22

May be related to https://github.com/influxdata/influxdb/issues/4089

andig commented 2 years ago

No feedback? This would impact anyone scraping prometheus metrics produced by a recent prometheus library?

Sniper91 commented 2 years ago

influxdb don't support NaN by design. When encoding float value, it uses NaN as a sentinel to mark the end of data block. https://github.com/influxdata/influxdb/blob/master/tsdb/engine/tsm1/float.go

andig commented 2 years ago

influxdb don't support NaN by design.

...and that exactly is the problem when using Influx to scrape prometheus. It seems though, that prometheus has reverted the change for enabling NaNs.