mem
commented
3 days ago After thinking about this more, I realized that the problem is that if we wait until the next time the check should have run, we might end up with gaps in the data.
We still have some situations where we can wait, but not all of them.
Currently checks start running based on when the agent received them, not when they were defined.
If the check is transferred to a different agent instance (Grafana Labs issue), or if the agent restarts (Grafana Labs issue + user-visible issue) this resets. This means that the check might end up running slightly more often than expected.
To fix this, look at the creation time (modification?) of the check and compute the next time the check should run. This would preserve the frequency of the runs.
The very first time the check runs is close to the creation time, but it's never going to be the creation time. For checks with low frequencies, this means the first time is "far" in the future.
If C is the creation time and R > C is the time the agent received the check definition, right now the check starts to run at R + random(2 minutes).
In order to get that first check running as soon as possible, we should look at R - C, and if that's less than some threshold, we should run the check immediately. Otherwise we wait until C + f, where f is the frequency found in the check's definition.