roidelapluie
commented
6 months ago Are you sure about the version of your public Prometheus server?
Increase () should NEVER return negative values.
Open klausenbusk opened 7 months ago
roidelapluie
commented
6 months ago Are you sure about the version of your public Prometheus server?
Increase () should NEVER return negative values.
machine424
commented
6 months ago Could it be due to stale markers not being migrated? I don’t expect increase() to rely on them at this point though.
I was able to reproduce this on 6fd7c0c297efd8a66b30e6b11cefc1f922f0fdf5 by the way, just make the initial prometheus scrapes a counter that will disappear and get reset from time to time.
klausenbusk
commented
6 months ago Are you sure about the version of your public Prometheus server?
Sorry for the confusion.
Promtool version 2.50.1 (branch: main, revision: eea6ab1cdd24ec69c94ba4b0d165030c89860c8b) was used for dumping in the openmetrics format, on our primary (private) Prometheus instance, as #13194 is not yet part of any released versions. For everything else (Prometheus on both servers and promtool on our "public" Prometheus instance) version 2.50.1 (branch: tarball, revision: 2.50.1) as packaged in Arch Linux was used.
What did you do?
On our primary (private) Prometheus instance (
promtool, version 2.50.1 (branch: main, revision: eea6ab1cdd24ec69c94ba4b0d165030c89860c8b))promtool tsdb create-blocks-from openmetrics openmetrics data --max-block-duration 168h(promtool, version 2.50.1 (branch: tarball, revision: 2.50.1))What did you expect to see?
The queries to behave the same way as on our primary (private) Prometheus instance.
What did you see instead? Under which circumstances?
The
increasequery function is behaving vastly different on our "public" Prometheus instance.For the
node_network_transmit_bytes_total{instance=~".*\\.mirror\\.pkgbuild\\.com",device=~"en.*"}query, the result is the same.Primary (private) Prometheus instance:
"Public" Prometheus instance:
For the
increase(node_network_transmit_bytes_total{instance=~".*\\.mirror\\.pkgbuild\\.com",device=~"en.*"}[30d])query, the result is vastly different.Primary (private) Prometheus instance:
"Public" Prometheus instance:
System information
Linux 6.7.9-arch1-1 x86_64
Prometheus version
Prometheus configuration file
From our primary (private) Prometheus instance.
Alertmanager version
Alertmanager configuration file
Logs
From our "public" Prometheus instance.