Inject 0 value if first point in range has timestamp greater than n*scrape_interval

[sonnyli]

Proposal

Hi, we're facing an issue where the first delta for a counter isn't being captured, and rate returns 0. Is there any issue with injecting a 0 value if the first point in a range has timestamp greater than n*scrape_interval?

[free]

There are many reasons why a counter may pop into existence with a non-zero value. The most common is indeed that the counter just got created and incremented in one go, but that's not always the case. It could be that Prometheus missed a few scrapes; or the target just got added; or the counter is persisted across process restarts; or the counter never actually restarts from zero (e.g. is an ever-increasing timestamp).

There is no way for Prometheus to tell those apart, so it takes the safe approach, which is to never assume a counter starts at zero.

The workaround, if those events are that important to you, is to explicitly initialize them with zero at process start. Then you'll (almost) never miss an increment.

[raags]

The workaround, if those events are that important to you, is to explicitly initialize them with zero at process start. Then you'll (almost) never miss an increment.

This doesn't work if by the time Prometheus scrapes the counter, the application increments it. The only solution seems to go for a push-based architecture to solve all of this staleness problems.