sysadmin1139
commented
7 months ago Full event-log for the failing 18:15 event.
logger=datasources rule_uid=_Nu3p9V4z org_id=1 t=2024-02-08T18:15:07.62330725Z level=debug msg="Querying for data source via SQL store" uid=S_Kqe5e7k orgId=1
logger=ngalert.state.manager rule_uid=_Nu3p9V4z org_id=1 t=2024-02-08T18:15:12.870653042Z level=debug msg="State manager processing evaluation results" resultCount=1
logger=ngalert.scheduler rule_uid=_Nu3p9V4z org_id=1 version=10 attempt=0 now=2024-02-08T18:15:00Z t=2024-02-08T18:15:12.870256927Z level=debug msg="Alert rule evaluated" results="[{Instance: State:NoData Error:<nil> Results:map[] Values:map[A0:{Var:A Labels: Value:<nil>}] EvaluatedAt:2024-02-08 18:15:00 +0000 UTC EvaluationDuration:12.870241593s EvaluationString:[ var='A0' metric='NoData' labels={} value=null ]}]" duration=5.247031669s
logger=expr rule_uid=_Nu3p9V4z org_id=1 datasourceType=grafana-timestream-datasource t=2024-02-08T18:15:12.870148906Z level=debug msg="expression datasource query (seriesSet)" query=B
logger=ngalert.state.manager rule_uid=_Nu3p9V4z org_id=1 t=2024-02-08T18:15:12.871205061Z level=debug msg="Saving alert states" count=1
logger=ngalert.state.manager rule_uid=_Nu3p9V4z org_id=1 t=2024-02-08T18:15:12.871138996Z level=debug msg="Setting alert state"
logger=ngalert.state.historian rule_uid=_Nu3p9V4z org_id=1 t=2024-02-08T18:15:12.874227359Z level=debug msg="Alert state changed creating annotation" newState="Normal (NoData)" oldState=Normal
logger=ngalert.state.historian rule_uid=_Nu3p9V4z org_id=1 t=2024-02-08T18:15:12.874466849Z level=error msg="Error saving alert annotation batch" error="json: error calling MarshalJSON for type *simplejson.Json: json: unsupported value: NaN"
logger=ngalert.state.historian rule_uid=_Nu3p9V4z org_id=1 t=2024-02-08T18:15:12.874597227Z level=debug msg="Done saving alert annotation batch"
iwysiu
katebrenner
What happened:
We updated to 2.9.1 last week, which was fine. Earlier this week we made some adjustments to our metrics ingestion which resulted in a larger volume entering Timestream tables used in alerting. Alarms hitting our most increased tables started seeing sporadic
NoDataerrors. Turning debug logging on didn't give any gun-smoke.The 18:15 evaluation triggered a NoData, where the 18:16 evaluation cleared it. This flips minute by minute, sometimes going as long as 6 minutes of success before another NoData.
The change in 9.5 to only fire NoData after the For period has elapsed would have obscured this from visibility.
This feels like something internal to the grafana process is getting overwhelmed.
What you expected to happen:
It should have behaved as it formerly did, doing alert queries without issue.
How to reproduce it (as minimally and precisely as possible):
Screenshots
Title and series-names are deliberately obscured, but this is charting SampleCount for a variety of Timestream tables. Pixel resolution in this scale is 5 minutes. The alert flapping issue started shortly after the increase in ingestion rate.
Anything else we need to know?:
Environment: