influxdata / kapacitor

Open source framework for processing, monitoring, and alerting on time series data
MIT License
2.31k stars 492 forks source link

Alert rule with retention policy doesn't trigger #2569

Open 12nick12 opened 3 years ago

12nick12 commented 3 years ago

Steps to reproduce: List the minimal actions needed to reproduce the behavior.

  1. Add alert rule with retention policy
  2. Rule is never triggered

Expected behavior: I expect the rule to trigger an alert

Actual behavior: An alert does not get triggered

Environment info: RHEL 7 Chronograf 1.8.10-1 InfluxDB: 1.8.4

sranka commented 3 years ago

Can you please share the TICKscript of your alert rule? Do you have other alert rules / tickscripts that work fine?

12nick12 commented 3 years ago

I used the alert rule builder and so far none of them seem to work on this system. I see devices under the

image

var db = 'servers' var rp = '90days' var measurement = 'disk' var groupBy = [] var whereFilter = lambda: isPresent("used_percent") var name = 'Disk Usage' var idVar = name var message = '{{ .ID }} is {{ .Level }} value:{{ index .Fields "value" }}, {{ if not .Recoverable }}non-recoverable{{ end }}' var idTag = 'alertID' var levelTag = 'level' var messageField = 'message' var durationField = 'duration' var outputDB = 'chronograf' var outputRP = 'autogen' var outputMeasurement = 'alerts' var triggerType = 'threshold' var details = '{{ .ID }} is {{ .Level }} value:{{ index .Fields "value" }}, {{ if not .Recoverable }}non-recoverable{{ end }}' var crit = 15 var data = stream |from() .database(db) .retentionPolicy(rp) .measurement(measurement) .groupBy(groupBy) .where(whereFilter) |eval(lambda: "used_percent") .as('value') var trigger = data |alert() .crit(lambda: "value" > crit) .message(message) .id(idVar) .idTag(idTag) .levelTag(levelTag) .messageField(messageField) .durationField(durationField) .details(details) .email() .to('email@example.com') trigger |eval(lambda: float("value")) .as('value') .keep() |influxDBOut() .create() .database(outputDB) .retentionPolicy(outputRP) .measurement(outputMeasurement) .tag('alertName', name) .tag('triggerType', triggerType) trigger |httpOut('output')

12nick12 commented 3 years ago

Any more info you'd like from me?

12nick12 commented 3 years ago

Anything else I can try?

12nick12 commented 3 years ago

Any potential reasons you can think of?