search

influxdata / kapacitor

Open source framework for processing, monitoring, and alerting on time series data
MIT License
2.32k stars 493 forks source link

kapacitor is alerting to slack #1383

Open ravitoait opened 7 years ago

ravitoait commented 7 years ago

Hi Team, I have below installed to my Mac OS: InfluxDB v1.2.3, Telegraf v1.2.1 and Kapacitor.

After running Influx DB, starting telegraf and starting kapacitor, I can see the alerts being logged to the log file present in my Mac but I can't see the alerts being sent to Slack channel. Below is output of the: kapacitor show cpu_alert2

ID: cpu_alert2
Error: 
Template: 
Type: stream
Status: enabled
Executing: true
Created: 16 May 17 16:38 CDT
Modified: 17 May 17 12:09 CDT
LastEnabled: 17 May 17 12:09 CDT
Databases Retention Policies: ["kapacitor_example"."autogen"]
TICKscript:
stream
    // Select just the cpu measurement from our example database.
    |from()
        .measurement('cpu')
    |alert()
        .crit(lambda: sigma("usage_idle") > 3)
        // Whenever we get an alert write it to a file.
        .log('/Users/ravikumarsingh/Desktop/alerts.log')
        .slack()

DOT:
digraph cpu_alert2 {
graph [throughput="0.00 points/s"];

stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="962"];

from1 [avg_exec_time_ns="619ns" ];
from1 -> alert2 [processed="962"];

alert2 [alerts_triggered="21" avg_exec_time_ns="72.024µs" crits_triggered="12" infos_triggered="0" oks_triggered="9" warns_triggered="0" ];
}

So, I can see 21 alerts being triggered and I can see the alerts to the log file also but its not being sent to the slack channel.

Below is the slack configuration in kapacitor.conf file:

[slack]
  enabled = true
  url = "https://hooks.slack.com/services/********/********/******"
  channel = "#general"
  username = "kapacitor"
  icon-emoji = ""
  global = true
  state-changes-only = false

Below is telegrapf.conf file:
[agent]
    interval = "1s"

[outputs]

# Configuration to send data to InfluxDB.
[outputs.influxdb]
    urls = ["http://localhost:8086"]
    database = "kapacitor_example"
    user_agent = "telegraf"

# Collect metrics about cpu usage
[cpu]
    percpu = false
    totalcpu = true
    drop = ["cpu_time"]
~

I am not sure if there is some issue with my Databases Retention Policies. Requesting you to please let me know what I am doing wrong here.

jaapz commented 7 years ago

You should probably remove that URL from the issue, I think everyone can post to your slack channel now

ravitoait commented 7 years ago

Thanks @jaapz! Can someone please tell me what I am doing wrong here. @nathanielc

nathanielc commented 7 years ago

@ravitoait Are there any error logs for Kapacitor?

ravitoait commented 7 years ago

@nathanielc Below is all I can see in the Kapacitor log, I dont see any error:

[run] 2017/05/18 11:21:51 I! Kapacitor starting, version v1.2.1, branch unknown, commit 4628bda2e3fbcc43df694707ec3f640f213cb0dc [run] 2017/05/18 11:21:51 I! Go version go1.8.1 [srv] 2017/05/18 11:21:51 I! Kapacitor hostname: localhost [srv] 2017/05/18 11:21:51 I! ClusterID: 9f3918eb-749d-473a-bbb0-7a792705808b ServerID: 625792ec-3838-4ce9-b776-44d5991ab322 [task_master:main] 2017/05/18 11:21:51 I! opened [srv] 2017/05/18 11:21:51 D! opening service: storage.Service [srv] 2017/05/18 11:21:51 D! opened service: storage.Service [srv] 2017/05/18 11:21:51 D! opening service: noauth.Service [srv] 2017/05/18 11:21:51 D! opened service: noauth.Service [srv] 2017/05/18 11:21:51 D! opening service: config.Service [srv] 2017/05/18 11:21:51 D! opened service: config.Service [srv] 2017/05/18 11:21:51 D! applying configuration overrides [srv] 2017/05/18 11:21:51 D! applying configuration overrides for opsgenie [srv] 2017/05/18 11:21:51 D! applying configuration overrides for snmptrap [srv] 2017/05/18 11:21:51 D! applying configuration overrides for sensu [srv] 2017/05/18 11:21:51 D! applying configuration overrides for slack [srv] 2017/05/18 11:21:51 D! applying configuration overrides for telegram [srv] 2017/05/18 11:21:51 D! applying configuration overrides for victorops [srv] 2017/05/18 11:21:51 D! applying configuration overrides for hipchat [srv] 2017/05/18 11:21:51 D! applying configuration overrides for pagerduty [srv] 2017/05/18 11:21:51 D! applying configuration overrides for smtp [srv] 2017/05/18 11:21:51 D! applying configuration overrides for talk [srv] 2017/05/18 11:21:51 D! applying configuration overrides for alerta [srv] 2017/05/18 11:21:51 D! applying configuration overrides for influxdb [srv] 2017/05/18 11:21:51 D! applying configuration overrides for kubernetes [srv] 2017/05/18 11:21:51 D! opening service: servicetest.Service [srv] 2017/05/18 11:21:51 D! opened service: servicetest.Service [srv] 2017/05/18 11:21:51 D! opening service: alert.Service [srv] 2017/05/18 11:21:51 D! opened service: alert.Service [srv] 2017/05/18 11:21:51 D! opening service: udf.Service [srv] 2017/05/18 11:21:51 D! opened service: udf.Service [srv] 2017/05/18 11:21:51 D! opening service: deadman.Service [srv] 2017/05/18 11:21:51 D! opened service: deadman.Service [srv] 2017/05/18 11:21:51 D! opening service: influxdb.Service [influxdb] 2017/05/18 11:21:51 D! linking subscriptions for cluster default [srv] 2017/05/18 11:21:51 D! opened service: influxdb.Service [srv] 2017/05/18 11:21:51 D! opening service: task_store.Service [task_store] 2017/05/18 11:21:51 D! could not open old boltd for task_store. Not performing migration. Remove the task_store.dir configuration to disable migration. [srv] 2017/05/18 11:21:51 D! opened service: task_store.Service [srv] 2017/05/18 11:21:51 D! opening service: replay.Service [srv] 2017/05/18 11:21:51 D! opened service: replay.Service [srv] 2017/05/18 11:21:51 D! opening service: alerta.Service [srv] 2017/05/18 11:21:51 D! opened service: alerta.Service [srv] 2017/05/18 11:21:51 D! opening service: hipchat.Service [srv] 2017/05/18 11:21:51 D! opened service: hipchat.Service [srv] 2017/05/18 11:21:51 D! opening service: opsgenie.Service [srv] 2017/05/18 11:21:51 D! opened service: opsgenie.Service [srv] 2017/05/18 11:21:51 D! opening service: pagerduty.Service [srv] 2017/05/18 11:21:51 D! opened service: pagerduty.Service [srv] 2017/05/18 11:21:51 D! opening service: smtp.Service [smtp] 2017/05/18 11:21:51 I! Starting SMTP service [srv] 2017/05/18 11:21:51 D! opened service: smtp.Service [srv] 2017/05/18 11:21:51 D! opening service: telegram.Service [srv] 2017/05/18 11:21:51 D! opened service: telegram.Service [srv] 2017/05/18 11:21:51 D! opening service: hipchat.Service [srv] 2017/05/18 11:21:51 D! opened service: hipchat.Service [srv] 2017/05/18 11:21:51 D! opening service: alerta.Service [srv] 2017/05/18 11:21:51 D! opened service: alerta.Service [srv] 2017/05/18 11:21:51 D! opening service: slack.Service [srv] 2017/05/18 11:21:51 D! opened service: slack.Service [srv] 2017/05/18 11:21:51 D! opening service: snmptrap.Service [srv] 2017/05/18 11:21:51 D! opened service: snmptrap.Service [srv] 2017/05/18 11:21:51 D! opening service: sensu.Service [srv] 2017/05/18 11:21:51 D! opened service: sensu.Service [srv] 2017/05/18 11:21:51 D! opening service: slack.Service [srv] 2017/05/18 11:21:51 D! opened service: slack.Service [srv] 2017/05/18 11:21:51 D! opening service: talk.Service [srv] 2017/05/18 11:21:51 D! opened service: talk.Service [srv] 2017/05/18 11:21:51 D! opening service: telegram.Service [srv] 2017/05/18 11:21:51 D! opened service: telegram.Service [srv] 2017/05/18 11:21:51 D! opening service: victorops.Service [srv] 2017/05/18 11:21:51 D! opened service: victorops.Service [srv] 2017/05/18 11:21:51 D! opening service: k8s.Service [srv] 2017/05/18 11:21:51 D! opened service: k8s.Service [srv] 2017/05/18 11:21:51 D! opening service: stats.Service [stats] 2017/05/18 11:21:51 I! opened service [srv] 2017/05/18 11:21:51 D! opened service: stats.Service [srv] 2017/05/18 11:21:51 D! opening service: reporting.Service [srv] 2017/05/18 11:21:51 D! opened service: reporting.Service [httpd] 2017/05/18 11:21:51 I! Starting HTTP service [httpd] 2017/05/18 11:21:51 I! Authentication enabled: false

[run] 2017/05/18 11:21:51 I! Listening for signals [httpd] ::1 - - [18/May/2017:11:21:53 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 1a9cd629-3be6-11e7-8001-000000000000 378 [httpd] ::1 - - [18/May/2017:11:22:00 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 1e9b40cc-3be6-11e7-8002-000000000000 750 [httpd] ::1 - - [18/May/2017:11:22:03 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 2093021b-3be6-11e7-8003-000000000000 178 [httpd] ::1 - - [18/May/2017:11:22:10 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 2490da02-3be6-11e7-8004-000000000000 856 [httpd] ::1 - - [18/May/2017:11:22:13 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 2689007b-3be6-11e7-8005-000000000000 243 [httpd] ::1 - - [18/May/2017:11:22:20 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 2a873399-3be6-11e7-8006-000000000000 1197 [httpd] ::1 - - [18/May/2017:11:22:23 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 2c7f05b0-3be6-11e7-8007-000000000000 246 [httpd] ::1 - - [18/May/2017:11:22:30 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 307ce15a-3be6-11e7-8008-000000000000 1235 [httpd] ::1 - - [18/May/2017:11:22:33 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 32749f2f-3be6-11e7-8009-000000000000 342 [httpd] ::1 - - [18/May/2017:11:22:40 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 3672c509-3be6-11e7-800a-000000000000 871 [httpd] ::1 - - [18/May/2017:11:22:43 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 386abe65-3be6-11e7-800b-000000000000 228 [httpd] ::1 - - [18/May/2017:11:22:50 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 3c68d0a0-3be6-11e7-800c-000000000000 1545 [influxdb] 2017/05/18 11:22:51 D! linking subscriptions for cluster default [httpd] ::1 - - [18/May/2017:11:22:53 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 3e605e1d-3be6-11e7-800d-000000000000 349 [httpd] ::1 - - [18/May/2017:11:23:00 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 425e7cf4-3be6-11e7-800e-000000000000 1415 [httpd] ::1 - - [18/May/2017:11:23:03 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 445651e7-3be6-11e7-800f-000000000000 158 [httpd] ::1 - - [18/May/2017:11:23:10 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 48540311-3be6-11e7-8010-000000000000 790 [httpd] ::1 - - [18/May/2017:11:23:13 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 4a4c4f57-3be6-11e7-8011-000000000000 158 [httpd] ::1 - - [18/May/2017:11:23:20 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 4e4a601d-3be6-11e7-8012-000000000000 1337 [httpd] ::1 - - [18/May/2017:11:23:23 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 5041db7a-3be6-11e7-8013-000000000000 213 [httpd] ::1 - - [18/May/2017:11:23:30 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 543fde6f-3be6-11e7-8014-000000000000 1233 [httpd] ::1 - - [18/May/2017:11:23:33 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 5637ab40-3be6-11e7-8015-000000000000 237 [httpd] ::1 - - [18/May/2017:11:23:40 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 5a3616b9-3be6-11e7-8016-000000000000 606 [httpd] ::1 - - [18/May/2017:11:23:43 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 5c2e0b6f-3be6-11e7-8017-000000000000 224 [httpd] ::1 - - [18/May/2017:11:23:50 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 602bd099-3be6-11e7-8018-000000000000 715 [influxdb] 2017/05/18 11:23:51 D! linking subscriptions for cluster default [httpd] ::1 - - [18/May/2017:11:23:53 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 6223f36d-3be6-11e7-8019-000000000000 273 [httpd] ::1 - - [18/May/2017:11:24:00 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 6621c002-3be6-11e7-801a-000000000000 663 [httpd] ::1 - - [18/May/2017:11:24:03 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 681953b9-3be6-11e7-801b-000000000000 245 [httpd] ::1 - - [18/May/2017:11:24:10 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 6c17deb9-3be6-11e7-801c-000000000000 1105 [httpd] ::1 - - [18/May/2017:11:24:13 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 6e0fb3eb-3be6-11e7-801d-000000000000 279 [httpd] ::1 - - [18/May/2017:11:24:20 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 720d7e8a-3be6-11e7-801e-000000000000 1098 [httpd] ::1 - - [18/May/2017:11:24:23 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 740544a8-3be6-11e7-801f-000000000000 176 [httpd] ::1 - - [18/May/2017:11:24:30 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" 78036228-3be6-11e7-8020-000000000000 1244 [httpd] ::1 - - [18/May/2017:11:24:33 -0500] "POST /write?consistency=&db=telegraf&precision=ns&rp=autogen HTTP/1.1" 204 0 "-" "InfluxDBClient" 79fb594b-3be6-11e7-8021-000000000000 183

nathanielc commented 7 years ago

When ever an alert is triggered there is a debug log with the text alert triggered and then a bunch of information. You should be seeing those logs, and since the slack integration is not working I would expect to see and error log shortly afterwards. Can you search specifically for the alert triggered logs and confirm there are no errors nearby?

ravitoait commented 7 years ago

@nathanielc No, I can't see any file with name "alert triggered" and even in the Kapacitor log I can't see "alert triggered" mentioned. But I can see alerts being recorded in the log file (file alerts.txt attached). Below is my kapacitor.conf file:

hostname = "localhost" data_dir = "/Users/ravikumarsingh/.kapacitor" skip-config-overrides = false default-retention-policy = ""

[http] bind-address = ":9093" auth-enabled = false log-enabled = true write-tracing = false pprof-enabled = false https-enabled = false https-certificate = "/etc/ssl/kapacitor.pem" shutdown-timeout = "10s" shared-secret = ""

[replay] dir = "/Users/ravikumarsingh/.kapacitor/replay"

[storage] boltdb = "/Users/ravikumarsingh/.kapacitor/kapacitor.db"

[task] dir = "/Users/ravikumarsingh/.kapacitor/tasks" snapshot-interval = "1m0s"

[[influxdb]] enabled = true name = "default" default = false urls = ["http://localhost:8086"] username = "" password = "" ssl-ca = "" ssl-cert = "" ssl-key = "" insecure-skip-verify = false timeout = "0s" disable-subscriptions = false subscription-protocol = "http" kapacitor-hostname = "" http-port = 0 udp-bind = "" udp-buffer = 1000 udp-read-buffer = 0 startup-timeout = "5m0s" subscriptions-sync-interval = "1m0s" [influxdb.excluded-subscriptions] _kapacitor = ["autogen"]

[logging] file = "/Users/ravikumarsingh/Desktop/kapacitor.log" level = "DEBUG"

[config-override] enabled = true

[alert]

[collectd] enabled = false bind-address = ":25826" database = "collectd" retention-policy = "" batch-size = 5000 batch-pending = 10 batch-timeout = "10s" read-buffer = 0 typesdb = "/usr/share/collectd/types.db"

[opentsdb] enabled = false bind-address = ":4242" database = "opentsdb" retention-policy = "" consistency-level = "one" tls-enabled = false certificate = "/etc/ssl/influxdb.pem" batch-size = 1000 batch-pending = 5 batch-timeout = "1s" log-point-errors = true

[alerta] enabled = false url = "" insecure-skip-verify = false token = "" environment = "" origin = ""

[hipchat] enabled = false url = "" token = "" room = "" global = false state-changes-only = false

[opsgenie] enabled = false api-key = "" url = "https://api.opsgenie.com/v1/json/alert" recovery_url = "https://api.opsgenie.com/v1/json/alert/note" global = false

[pagerduty] enabled = false url = "https://events.pagerduty.com/generic/2010-04-15/create_event.json" service-key = "" global = false

[smtp] enabled = true host = "localhost" port = 25 username = "" password = "" from = "kapacitor@example.com" to = ["ravitoait@gmail.com"] no-verify = false global = true state-changes-only = false idle-timeout = "30s"

[snmptrap] enabled = false addr = "localhost:162" community = "kapacitor" retries = 1

[sensu] enabled = false addr = "" source = "Kapacitor"

[slack] enabled = true url = "https://hooks.slack.com/services/****/****/***" channel = "#general" username = "kapacitor" icon-emoji = "" global = true state-changes-only = false

[talk] enabled = false url = "" author_name = ""

[telegram] enabled = false url = "https://api.telegram.org/bot" token = "" chat-id = "" parse-mode = "" disable-web-page-preview = false disable-notification = false global = false state-changes-only = false

[victorops] enabled = false api-key = "" routing-key = "" url = "https://alert.victorops.com/integrations/generic/20131114/alert" global = false

[kubernetes] enabled = false in-cluster = false token = "" ca-path = ""

[reporting] enabled = true url = "https://usage.influxdata.com"

[stats] enabled = true stats-interval = "10s" database = "_kapacitor" retention-policy = "autogen" timing-sample-rate = 0.1 timing-movavg-size = 1000

[udf]

[deadman] interval = "10s" threshold = 0.0 id = "{{ .Group }}:NODE_NAME for task '{{ .TaskName }}'" message = "{{ .ID }} is {{ if eq .Level \"OK\" }}alive{{ else }}dead{{ end }}: {{ index .Fields \"emitted\" | printf \"%0.3f\" }} points/INTERVAL." global = false

ravitoait commented 7 years ago

@nathanielc any help would be highly appreciated!

nathanielc commented 7 years ago

@ravitoait I am not sure what the issue is. Without any error logs or other information its hard to debug. Version 1.3.0 was released this week, maybe you could try upgrading to see if the issue is resolved? Your configuration looks good.

ravitoait commented 7 years ago

@nathanielc where exactly (file path) I can find error logs. I am using Mac machine. I am using below config for log: [logging] file = "/Users/ravikumarsingh/Desktop/kapacitor.log" level = "DEBUG" and I dont see any error in the file /Users/ravikumarsingh/Desktop/kapacitor.log.

Additionally, did you mean that whenever alert is triggered, a file with a name "alert triggered" would be generated and I should see some error message(since my slack is not working) in the file "alert triggered"?

nathanielc commented 7 years ago

The string alert triggered should be present in the file /Users/ravikumarsingh/Desktop/kapacitor.log. In that same file there should be many other logs. The level of the log is indicated by a D! for debug and a E! for error logs. All logs of any level are in the /Users/ravikumarsingh/Desktop/kapacitor.log file.

You can use this grep command to find what I am looking for. (I am not on a Mac so it might be slightly different but hopefully not)

grep -C 20 'alert triggered' /Users/ravikumarsingh/Desktop/kapacitor.log  | grep -P '(D!|E!)'

NOTE: The alert log file '/Users/ravikumarsingh/Desktop/alerts.log' specified in your TICKscript will only contain alert data. It will not contain any application logs from Kapacitor.

ravitoait commented 7 years ago

@nathanielc this is very weird, as I can't see any string "alert triggered" in the /Users/ravikumarsingh/Desktop/kapacitor.log though I can see so many alerts being generated.

Ravis-MacBook-Pro:AlertUsingKapacitor ravikumarsingh$ grep -C 20 'alert triggered' /Users/ravikumarsingh/Desktop/kapacitor.log Ravis-MacBook-Pro:AlertUsingKapacitor ravikumarsingh$

Ravis-MacBook-Pro:AlertUsingKapacitor ravikumarsingh$ kapacitor show cpu_alert4 ID: cpu_alert4 Error: Template: Type: stream Status: enabled Executing: true Created: 30 May 17 17:01 CDT Modified: 01 Jun 17 10:59 CDT LastEnabled: 01 Jun 17 10:59 CDT Databases Retention Policies: ["kapacitor_example"."autogen"] TICKscript: stream |from() .measurement('cpu') |alert() // Compare values to running mean and standard deviation .crit(lambda: sigma("usage_idle") > 3) .log('/Users/ravikumarsingh/Desktop/alerts.log') .slack() .channel('#iot-cloud-dev-ops')

DOT: digraph cpu_alert4 { graph [throughput="0.00 points/s"];

stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ]; stream0 -> from1 [processed="610"];

from1 [avg_exec_time_ns="509ns" errors="0" working_cardinality="0" ]; from1 -> alert2 [processed="610"];

alert2 [alerts_triggered="19" avg_exec_time_ns="334.282µs" crits_triggered="10" errors="0" infos_triggered="0" oks_triggered="9" warns_triggered="0" working_cardinality="1" ]; }

I am attaching /Users/ravikumarsingh/Desktop/kapacitor.log for your reference. Please have a look kapacitor.txt

ravitoait commented 7 years ago

@nathanielc another thing which I noticed, when I changed the level to ERROR, I can see so many below message in the kapacitor..log:

[httpd] ::1 - - [01/Jun/2017:11:26:00 -0500] "POST /write?consistency=&db=_internal&precision=ns&rp=monitor HTTP/1.1" 204 0 "-" "InfluxDBClient" ff709a43-46e6-11e7-8002-000000000000 1102

Are these errors?

ravitoait commented 7 years ago

@nathanielc Finally using Chronograf, I got the alert integrated with Slack, but while integrating with email, I am getting below errors( we are using SendGrid as SMTP server) [smtp] 2017/06/08 18:22:12 E! gomail: could not send email 1: mail: no address

Below is my Kapacitor.conf for SMTP: [smtp] enabled = true host = "smtp.sendgrid.net" port = 587 username = "" password = "**" no-verify = false global = true state-changes-only = false from = "@mycompanydomainname.com" to = ["myusername@mycompanydomainname.com"] idle-timeout = "30s"

Please help @nathanielc