search

jacksontj / promxy

An aggregating proxy to enable HA prometheus
MIT License
1.12k stars 126 forks source link

Promxy uses /api/v1/series for ALERTS_FOR_STATE #650

Open perederyaev opened 2 months ago

perederyaev commented 2 months ago

Hi!

Looks like restoring alert's state feature dosn't work. Promxy uses /api/v1/series for ALERTS_FOR_STATE it can't restore alert's state because result from /api/v1/series contains no timestamp.

Tested on most recent versions and current master with prometheus and victoriametrics as backends.

jacksontj commented 1 month ago

Thanks for opening an issue! I spent some time attempting to reproduce the error you are describing but I am unable to do so. When I run locally using the demo_robust.conf configuration and the rules.alertbackfill CLI flag it works fine locally (I see the query happening; and the alert goes to red within ~5s of startup.

I did some digging and I also can't see how a backfill query would hit the /series endpoint either. If we look at the backfill code we can see that it is doing a QueryRange which doesn't get converted to anything besides a query range all the way down.

So at this point I am unable to reproduce your reported issue; if you could provide more detail so that I can reproduce the issue; or some additional logs/errors/etc. that would go a long way towards getting a repro case.

perederyaev commented 1 month ago

Hi, the issue is occurring when rules.alertbackfill option is not specified. As I understand, rules.alertbackfill helps when no ALERTS_FOR_STATE available, but I have correct ALERTS_FOR_STATE metric and just want standard prometheus alert's restoration feature to work.