influxdata / kapacitor

Open source framework for processing, monitoring, and alerting on time series data
MIT License
2.32k stars 493 forks source link

stateChangeOnly triggers more alerts for each state #2084

Open buhlj opened 6 years ago

buhlj commented 6 years ago

Hello

InfluxDB: 1.6.4 Chronograf: 1.6.2 (git: 8ba29804dd3e52859363851b3fe73abab442dd0e) Kapacitor: OSS 1.5.1 (git: HEAD 89828ffff6cf5cd4cb2b34bf883e134395f734de)

I'm quite new to kapacitor and Chronograf but I have tried to setup and high flow alarm which should only trigger once per state change. The problem I'm facing is that even though the ".stateChangeOnly()" have been set on the Alert node then I still get multiple alerts for each state. I do not get for all the metrics values but for around 80% so some are sorted out.

The image shows all alerts and the actual data from my influx measurement. I would have expected only to get the two alerts marked with red.

alerts and metrics

Could this have something to do with the bulk writing to influx? that multiple messages are write in one write request?

And here is my kapacitor TICK script:

var db = 'OPCUA'

var rp = 'autogen'

var measurement = 'water0.flow.hot'

var groupBy = []

var whereFilter = lambda: TRUE

var name = 'Varmt vand forbrug HH'

var idVar = name

var message = 'Vand forbrug udenfor er højt, lige nu !!! '

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var outputDB = 'chronograf'

var outputRP = 'autogen'

var outputMeasurement = 'alerts'

var triggerType = 'threshold'

var crit = 4

var data = stream
    |from()
        .database(db)
        .retentionPolicy(rp)
        .measurement(measurement)
        .groupBy(groupBy)
        .where(whereFilter)
    |eval(lambda: "value")
        .as('value')

var trigger = data
    |alert()
        .crit(lambda: "value" > crit)
        .message(message)
        .id(idVar)
        .idTag(idTag)
        .levelTag(levelTag)
        .messageField(messageField)
        .durationField(durationField)
        .stateChangesOnly()

trigger
    |eval(lambda: float("value"))
        .as('value')
        .keep()
    |influxDBOut()
        .create()
        .database(outputDB)
        .retentionPolicy(outputRP)
        .measurement(outputMeasurement)
        .tag('alertName', name)
        .tag('triggerType', triggerType)

trigger
    |httpOut('output')
Gaurav-Pande commented 5 years ago

I am facing the similar issue. Were you able to find the solution to above. My kapacitor is also receiving points from the influxdb, but the kapacitor is generating multiple alerts on the same state instead of using the stateChangesOnly() function.

PMehrfeld commented 4 years ago

Same here 🤚 @buhlj or @Gaurav-Pande : Did you find any solution?

ConstipatedNinja commented 4 years ago

I'm also running into this problem. It seems that it will alert again if the measured value changes, even if the state hasn't changed. I don't know if this is accurate, but it seems to be what's happening.