search

timescale / pg_prometheus

PostgreSQL extension for Prometheus data
Apache License 2.0
213 stars 44 forks source link

Strange behaviour of queries with 'irate' function #14

Open maciejprz opened 6 years ago

maciejprz commented 6 years ago

screenshot-2018-7-17 grafana - report parser2 screenshot-2018-7-17 grafana - rabbitmq screenshot-2018-7-17 grafana1 - rabbitmq

Hello, We are using prometheus-postgresql-adapter and pg-prometheus services for keeping prometheus metrics in database storage. I'm not sure which one is causing the problem, but after switching from prometheus local persistent storage to those two services, all queries containing 'irate' function seem to behave incorrectly. It is especially visible in Grafana dashboards - graphs with 'irate' are showing 'no data' error, or there are no charts visible, only values are shown, or charts are visible only for ~2 seconds, and they disappear again. After changing 'irate' to 'rate' function there are no problems and everything works just fine.

niksajakovljevic commented 5 years ago

Hi @maciejprz,

In order to dig deeper into no data error I would recommend to run prometheus-postgresql-adapter with --log.level=debug and in the log file you will see SQL queries that are running against the TimescaleDB to fetch Prometheus data. You can test those SQL queries explicitly to make sure that they do return the data. The irate behaviour seems strange. The remote storage integration is only responsible for moving data back and forth between Prometheus instance and remote storage so shouldn't have anything to do with the PromQL(eg. irate).

Let me know if looking into SQL queries has helped.