InfluxDB v2 - E-Mail - Notification Endpoint

[StefanSa]

Proposal: Add E-Mail as Endpoint Notification.

Current behavior: No exist this option currently

Desired behavior: I want to enable notifications of alerts and send them to a E-Mail user / group.

Alternatives considered: No alternatives are considered so far.

Use case: This feature exists in Cronograf and I use it. That be great have in InfluxDB v2.

[emexelem]

I just upgraded to InfluxDB v2 and started to use it for monitoring and sending alert, but it seems there's no support for email, except for mail services having HTTP API, as described in https://docs.influxdata.com/influxdb/v2.0/monitor-alert/send-email/ Sending email through SMTP is a must have in my case. Do you have any idea when this will be supported?

[mohammadali-seifkashani]

Whick files should be edited and what functions should be implemented? I'm a beginner.

[StefanSa]

Hi there, @hoorayimhelping Any ETA for release of this feature?

[timhallinflux]

This is the recommended approach for now: https://docs.influxdata.com/influxdb/v2.0/monitor-alert/send-email/

[mirkocomparetti-synesis]

Thank you @timhallinflux The actual approach does not work for us and we really miss the standard smtp functionality. Hopefully it will make it to the release soon!

[yorik]

Would it be possible to call custom shell script or similar for now to send alerts? Using commercial mailing services wouldn't make sense if there is existing email server.

[haavee]

Actually, we were considering to start to use InfluxDB but then I checked whether sending email via SMTP on alert was possible (a strict requirement for our environment) and I couldn't find it in v2 - much to my surprise. This alone removes Influx from our list of candidates. If a shell-script-on-alert would be an endpoint, that would work fine as well.

[timhallinflux]

@haavee as mentioned here.... there are many ways to integrate with email providers to send email-based alerts. See examples listed here: https://docs.influxdata.com/influxdb/v2.0/monitor-alert/send-email/

You can also create your own call out using the HTTP post mechanism: https://docs.influxdata.com/influxdb/v2.0/reference/flux/stdlib/http/

[jdstrand]

@timhallinflux - while the examples you listed in https://docs.influxdata.com/influxdb/v2.0/monitor-alert/send-email/ are great, they all depend on sending email over HTTP rather than the SMTP protocol. Sites running OSS may have local SMTP servers for compliance or other reasons (specifically, folks may not want to/can't send their email through a 3rd party like Sendgrid, AWS SES, Mailjet or Mailgun). I'm unaware of a mature, open source HTTP email API that people can integrate into their infrastructure (I would love to be corrected on this).

The callout mechanism in https://docs.influxdata.com/influxdb/v2.0/reference/flux/stdlib/http/ could be used, but AIUI people then need to roll their own HTTP to SMTP solution (or HTTP to some other alerting mechanism) which seems like a high bar for something as simple as sending an alert to an smtp host as a client.

SMTP support has been standard for monitoring software for a long time and adding this support to OSS v2 would be great and help people who want to migrate from 1.x to 2.0 (indeed this is a feature gap between 1.x TICK and OSS v2). This should not be terribly difficult since there is a standard golang module for this (https://pkg.go.dev/net/smtp) and current kapacitor already has this support (https://docs.influxdata.com/kapacitor/v1.6/event_handlers/email/ and https://github.com/influxdata/kapacitor/blob/master/services/smtp/service.go#L109) using gomail.v2 (https://pkg.go.dev/gopkg.in/gomail.v2).

[jdstrand]

SMTP support has been standard for monitoring software for a long time and adding this support to OSS v2 would be great and help people who want to migrate from 1.x to 2.0 (indeed this is a feature gap between 1.x TICK and OSS v2).

One thing that I hadn't considered is that people migrating from 1.x TICK to 2.0 can continue to use kapacitor for SMTP if they need to (https://docs.influxdata.com/influxdb/v2.0/tools/kapacitor/). As part of a migration strategy, this is useful because people with existing TICKscripts can continue to use them (and the SMTP endpoint).

(Note: while newer kapacitor supports writing flux, you can only send alerts using flux notification endpoints (ie, (currently) the HTTP ones; besides, this isn't the same as using native v2 tasks anyway)).

[nagelp]

Not much happening regarding SMTP capability of InfluxDB v2.0, it seems. I suppose it wouldn't be too hard to write an adapter that listens on HTTP and hands off the message to localhost's smtpd. That way we could use InfluxDB's HTTP notification endpoint (@jdstrand suggested something similar more than a year ago). Maybe someone already wrote something like that? I'd be interested :)

[sawo1337]

@haavee as mentioned here.... there are many ways to integrate with email providers to send email-based alerts. See examples listed here: https://docs.influxdata.com/influxdb/v2.0/monitor-alert/send-email/

You can also create your own call out using the HTTP post mechanism: https://docs.influxdata.com/influxdb/v2.0/reference/flux/stdlib/http/

I honestly thought you were joking for a few moments there. SMTP is still an industry standard AFAIK. Not having that is an enormous disadvantage, to say the least. Using HTTP and other protocols to substitute SMTP is not a real solution.

[ndwarshuis]

As a workaround, writing an HTTP->SMTP proxy isn't all that hard.

Here's a super hacky python flask app that will take a POST request to http://somehost/send and sends an email through the indicated SMTP server (using AUTH).

from flask import Flask, jsonify, request
from flask_restful import Resource, Api
from flask_restful import reqparse
from smtplib import SMTP, SMTPException
from email.message import EmailMessage

app = Flask(__name__)
api = Api(app)

parser = reqparse.RequestParser()
parser.add_argument("host", type=str)
parser.add_argument("port", type=int)
parser.add_argument("tls", type=bool)
parser.add_argument("username", type=str)
parser.add_argument("password", type=str)
parser.add_argument("from", type=str)
parser.add_argument("to", type=str)
parser.add_argument("subject", type=str)
parser.add_argument("content", type=str)

def mk_msg(toaddr, fromaddr, subj, content):
    msg = EmailMessage()
    msg["To"] = toaddr
    msg["From"] = fromaddr
    msg["Subject"] = subj
    if content is not None:
        msg.set_content(content)
    return msg

class SendEmail(Resource):
    def post(self):
        args = parser.parse_args(strict=True)

        for key, msg in [
            ("host", "no host provided"),
            ("username", "no username provided"),
            ("to", "no 'to' address"),
            ("from", "no 'from' address"),
        ]:
            if args[key] is None:
                return msg, 400

        # just assume we will want TLS (for obvious reasons)
        port = 587 if args["port"] is None else args["port"]
        use_tls = args["tls"] is not False

        try:
            with SMTP(host=args["host"], port=port) as s:
                if use_tls:
                    s.starttls()
                s.login(args["username"], args["password"])
                msg = mk_msg(
                    args["to"],
                    args["from"],
                    args["subject"],
                    args["content"],
                )
                s.send_message(msg)
                return "send successful", 201
        except SMTPException as e:
            return str(e), 400

api.add_resource(SendEmail, "/send")

if __name__ == "__main__":
    app.run(debug=True)

Run it in docker or however you like.

Then adapt one of the flux tasks in the documentation to use the API (assuming the flask app is on localhost):

import "http"
import "json"
import "testing"

option task = {name: "SMTP Critical Alert", every: 1h}

// create API request data
postData = (r) =>
    ({
        "from": "a@domain.bla",
        "to": "b@domain.bla",
        "host": "smtp.domain.bla",
        "username": "blablabla",
        "password": "notanuclearcode",
        "subject": "InfluxDB critical alert",
        "content": "Check ${r._check_name} on ${r.host} had ${r._value} critical checks.",
    })

// send alert if counts are non-zero, return HTTP status code (201 == success)
sendMaybe = (r) =>
    if r._value > 1 then
        {r with _value:
                http.post(
                    url: "http://localhost:5000/send",
                    headers: {"Content-Type": "application/json"},
                    data: json.encode(v: postData(r)),
                ),
        }
    else
        {r with _value: 201}

from(bucket: "_monitoring")
    |> range(start: -task.every)
    |> filter(fn: (r) => r._measurement == "statuses" and r._level == "crit")
    |> group(columns: ["host", "_check_name"])
    |> count()
    |> map(fn: sendMaybe)
    // filter out success statuses
    |> filter(fn: (r) => r._value != 201)
    // puke if any return statuses are not 201
    |> testing.assertEmpty()

This seems to work...however, it doesn't seem like much to ask for something so simple to be in influxdb itself ;)