Alert per value in field of groupBy

[ejether]

Is there a way to have separate alerts base on a single tick for different fields of data?

Example:

For each of 'typeNames' show, I'd like there to be an individual alert on the media of each 'typeName' instead of an alert, in this case, based on the median 'queueSize' of all of the types selected?

I'd like this so I don't have to create a separate tick for each of the "typeNames" that I have.

    ::kapacitor::task::define{
        'general_q_size':
        influx_database => 'jmxtrans',
        influx_retention => 'default',
        tickfile_content => "var qsize = stream
        |from()
            .measurement('activemq.queue')
            .where(lambda: \"attributeName\" == 'QueueSize')
            .groupBy("typeName")
        |window()
            .period(5m)
            .every(1m)
        |median('QueueSize').as('median')
        |alert()
          .crit(lambda: \"median\" > 100)
          .log('/var/log/kapacitor/alerts.log')
          .alerta()
            .id('{{ index .Tags \"hostname\"}}_{{ index .Tags \"typeName\"}}')
            .resource('{{ index .Tags \"typeName\"}}')
            .value('{{ index .Fields \"median\"}}')
            .environment('Production')
            .details('Investigate elevated Queue {{ index .Tags \"typeName\"}} on {{ index .Tags \"hostname\"}}')
            .message('Investigate elevated Queue {{ index .Tags \"typeName\"}} on {{ index .Tags \"hostname\"}}')"
    }

which produces this data:

{
    "id": "dryrun-prramq-c01-n02.smarshinc.com_",
    "message": "Investigate elevated Queue!",
    "details": "Investigate elevated Queue",
    "time": "2016-09-21T22:01:22.512Z",
    "duration": 1559986000000,
    "level": "CRITICAL",
    "data": {
        "series": [{
            "name": "activemq.queue",
            "tags": {
                "hostname": "dryrun-prramq-c01-n02.smarshinc.com"
            },
            "columns": ["time", "QueueSize", "_jmx_port", "attributeName", "className", "objDomain", "typeName"],
            "values": [
                ["2016-09-21T22:01:22.512Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:02:22.496Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:02:22.506Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:02:22.512Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:03:22.491Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:03:22.499Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:03:22.506Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:04:22.495Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:04:22.513Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:04:22.524Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:05:22.498Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:05:22.501Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:05:22.504Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:06:22.49Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:06:22.511Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:06:22.513Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:07:22.504Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:07:22.512Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:07:22.52Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:08:22.491Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:08:22.5Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:08:22.514Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:09:22.494Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:09:22.503Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:09:22.511Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:10:22.493Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:10:22.494Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:10:22.495Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"],
                ["2016-09-21T22:11:22.492Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.dupes"],
                ["2016-09-21T22:11:22.496Z", 0, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=com.smarsh.prereviewer.filewatcher.requests.normal.deduped"],
                ["2016-09-21T22:11:22.498Z", 2533, 11099, "QueueSize", "org.apache.activemq.broker.jmx.QueueView", "org.apache.activemq", "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ"]
            ]
        }]
    }
}

[nathanielc]

@ejether I am not clear on what you are asking. Can you share the output of kapacitor show task for one of your tasks?

Based on what you shared it looks like you are templating the TICKscript with some external values but I am not quite sure I follow what the final result looks like.

[ejether]

Thanks for responding, I have since changed the Tick so the output of what your requesting would no longer make sense. Unfortunatly I didn't save the output of the previous tick either.

I'll see if I can do a little better job explaining it.

The JSON blob shown is the output to the log file in the .log node in the tick. In the data values section of the json blob shown there "typeName" column, one of the .groupBy in the .from node, there are three distinct "typeNames"

I'd like it if each type, based on the value, could raise a separate alert. For example here, the row with type "type=Broker,brokerName=localhost,destinationType=Queue,destinationName=ActiveMQ.DLQ" has a value of 2533, and the other two have a value of 0. If my alert threshold was value > 1 then the alert would trigger for just one type instead of as a combination of the type.

I want this because I want to be able to use a more generic tick instead of having a separate tick for each typeName

I hope that is more clear

[nathanielc]

@ejether If I understood correctly then you should be able to use an |eval() node to convert the field into a tag and then groupBy that tag to get separate alerts.

[ejether]

Great! I'll give that a try. Thanks

[SS30723]

@nathanielc .. stream |from() . where(lambda : (("application" == 'A' OR "application" == 'B') AND "reasoncode" == '404')) .groupBy('application') |window() .period(5m) .every(5m) |count('reasoncode') .as('value') |log() |alert() .crit(lambda: (value >2)) .log(/tmp/alert.log)

Question: In the alert.log, we want to the see the count based on the application, how to achieve that? currently in the alert.log, we see the output as application A and value as 10 (may be it is picking up the application name based on the first point received) But we are expecting to see the list of applications and their corresponding 404 values as per the above query.

[ashishkaransingh]

It would be great to have seperate alerts with multiple fields using on tickscript.

[ashishkaransingh]

@ejether did you get what you were looking for based on nathanielc's suggestion?

[ejether]

@ashishkaransingh it has been a few years. I am no longer using Influx and can't easily answer your question. Sorry!