search

VictoriaMetrics / VictoriaMetrics

VictoriaMetrics: fast, cost-effective monitoring solution and time series database
https://victoriametrics.com/
Apache License 2.0
10.96k stars 1.1k forks source link

Strangeness with `timestamp()` #6219

Open lapo-luchini opened 2 weeks ago

lapo-luchini commented 2 weeks ago

Describe the bug

If I use this I get all the results:

vm_app_version{job="vmagent"}

but when I do this I get a varying amount of records:

timestamp(vm_app_version{job="vmagent"})

I use this kind of rule to detect when a remote vmagent stopped responding, but I am now getting false positives.

I noticed that this happened when I upgraded vmagent from 1.93.x to 1.100.1, and not when (previously) I upgraded victoria-metrics di 1.100.1.

To Reproduce

timestamp

Version

victoria-metrics-20240411-091522-tags-v1.100.1-0-g9ee51e3

Logs

No response

Screenshots

No response

Used command-line flags

No response

Additional information

No response

lapo-luchini commented 1 week ago

Seems to be fixed in v1.101.0… ? (works with Grafana Explore, but the VMUI interface is broken — might be an issue with the fact that I built it myself though, I didn't check in deeper yet)

hagen1778 commented 4 days ago

Hello @lapo-luchini! timestamp function checks for the last timestamp over specified look-behind window timestamp(foo[5m]). If look-behind window is omitted timestamp(foo), then it uses step param instead. On the screenshot you have step 5s, so result is likely depends on whether you had a successful scrape during last 5s + latency_offset (30s), or not. Grafana Explore probably doesn't send step param, so it defaults to 5m when omitted. Does it make sense?

lapo-luchini commented 4 days ago

Grafana Explore probably doesn't send step param, so it defaults to 5m when omitted. Does it make sense?

It does! Originally I noticed the problem in vmalert rules, tough, not using vmui. It had a lot of false negatives in latest updates that didn't happen before. Might have been a change… but explanation makes sense, I'll add an explicit look-behind window to the rule.