alerta / alerta-contrib

Contributed integrations, plugins and custom webhooks
http://alerta.io
MIT License
119 stars 171 forks source link

Mailer seems to be working but no email are being sent #328

Closed alexd124641 closed 2 years ago

alexd124641 commented 4 years ago

Issue Summary I've installed alerta (which work fine) and alerta-mailer. When I do alerta-mailer (or execute mailer.py), everything seems to be working (connection to the database and heartbeat) I can see the heartbeat in the web UI but I don't receive any email.

Environment

To Reproduce Steps to reproduce the behavior:

  1. Execute alerta-mailer
  2. Generate new alerts after the POST command (which results in 201)
  3. No email are being sent (nothing in the logs)

Expected behavior At least having a trace somewhere of the email in order to identify what is going wrong

Alerta config

Alerta.conf:

[DEFAULT]
endpoint = http://localhost:8080/api
key = 6Xric-orNcEnCJZ6A5MVVQwdDKVw-Wc6JztIvLLO

[alerta-mailer]
key = 6Xric-orNcEnCJZ6A5MVVQwdDKVw-Wc6JztIvLLO
mail_to = mailhog@mailhog.example
mail_from = alerta@mailhog.example
amqp_url = mongodb://localhost:27017/kombu
endpoint = http://localhost:8080/api
dashboard_url = http://localhost:8000
smtp_host = localhost
smtp_port = 1025
smtp_use_ssl = False
debug = True
skip_mta = False
email_type = text
amqp_topic = notify
config_file = /etc/alerta/alerta.conf

alertad.conf:

#!/usr/bin/bash
PLUGINS = ['remote_ip','reject','heartbeat','blackout', 'prometheus', 'alerta-mailer', 'amqp']
ALERTA_TIMEOUT = 14400
ALERT_TIMEOUT = 14400
SECRET_KEY = '6Xric-orNcEnCJZ6A5MVVQwdDKVw-Wc6JztIvLLO'
BASE_URL = '/api'
DEBUG = True
LOG_HANDLERS = ['console', 'file']
LOG_FILE = '/var/log/alerta/alertad.log'
LOG_MAX_BYTES = 5*1024*1024  # 5 MB
LOG_BACKUP_COUNT = 2
LOG_FORMAT = 'verbose'

DATABASE_URL = 'mongodb://127.0.0.1:27017/?replicaSet=test&connectTimeoutMS=300000'
DATABASE_NAME = 'monitoring'
DATABASE_RAISE_ON_ERROR = False

SEVERITY_MAP = {
    'security': 0,
    'critical': 1,
    'major': 2,
    'minor': 3,
    'warning': 4,
    'indeterminate': 5,
    'normal': 9,
    'ok': 9,
    'unknown': 9,
    'informational': 9,
}

DEFAULT_NORMAL_SEVERITY = 'ok'  # 'normal', 'ok', 'cleared'
DEFAULT_PREVIOUS_SEVERITY = 'indeterminate'

COLOR_MAP = {
    'severity': {
        'security': '#B3B3B3',
        'critical': 'red',
        'major': 'orange',
        'minor': '#9B870C',
        'warning': '#1E90FF',
        'indeterminate': 'lightblue',
        'normal': '#00CC00',
        'ok': '#00CC00',
        'unknown': 'silver',
        'informational': 'grey'
    },
    'text': 'black',
    'highlight': 'skyblue '
}

EMAIL_VERIFICATION = False
MAIL_FROM = 'alerta@mailhog.example'
SMTP_HOST = 'localhost'
SMTP_PORT = 1025
SMTP_STARTTLS = False

Mailer.py :

DEFAULT_OPTIONS = {
    'config_file':   '/etc/alerta.conf',
    'profile':       'alertamailer',
    'endpoint':      'http://127.0.0.1:8080/api',
    'key':           '6Xric-orNcEnCJZ6A5MVVQwdDKVw-Wc6JztIvLLO',
    'amqp_url':      'mongodb://127.0.0.1:27017/kombu',
    'amqp_topic':    'notify',
    'amqp_queue_name':    '', # Name of the AMQP queue. Default is no name (default queue destination).
    'amqp_queue_exclusive': True, # Exclusive queues may only be consumed by the current connection.
    'smtp_host':     '127.0.0.1',
    'smtp_port':     1025,
    'smtp_username': '', # application-specific username if it differs from the specified 'mail_from' user
    'smtp_password': '',  # application-specific password if gmail used
    'smtp_starttls': True,  # use the STARTTLS SMTP extension
    'smtp_use_ssl': False,  # whether or not SSL is being used for the SMTP connection
    'ssl_key_file': None, # a PEM formatted private key file for the SSL connection
    'ssl_cert_file': None, # a certificate chain file for the SSL connection
    'mail_from':     '',  # alerta@example.com
    'mail_to':       [],  # devops@example.com, support@example.com
    'mail_localhost': 'mailhog.example',  # fqdn to use in the HELO/EHLO command
    'mail_template':  os.path.dirname(__file__) + os.sep + 'email.tmpl',
    'mail_template_html': os.path.dirname(__file__) + os.sep + 'email.html.tmpl',  # nopep8
    'mail_subject':  ('[{{ alert.status|capitalize }}] {{ alert.environment }}: '
                      '{{ alert.severity|capitalize }} {{ alert.event }} on '
                      '{{ alert.service|join(\',\') }} {{ alert.resource }}'),
    'dashboard_url': 'http://127.0.0.1:3000',
    'debug':         True,
    'skip_mta':      False,
    'email_type':    'html'  # options are: text, html
}

Output of either ./mailer.py or alerta-mailer:

root@112501L01SUP:/etc/alerta/mailer# ./mailer.py
DEBUG:__main__:Looking for rules files in /etc/alerta/alerta.rules.d
INFO:kombu.mixins:Connected to mongodb://localhost:27017/kombu
Connected to mongodb://localhost:27017/kombu
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8080
Starting new HTTP connection (1): localhost:8080
DEBUG:urllib3.connectionpool:http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 454
http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 454

If you need any additional info, file or command output let me know.

decentral1se commented 3 years ago

Hey, having the same issue. I see that 'alerta-mailer' shouldn't be in PLUGINS in your alertad.conf because it is not a plugin. That's one thing. Otherwise, I do not know, I have the exact same issue and I am not getting any logging at all. I suspect the logic is not even running because there are plenty of debug statements in the code...

decentral1se commented 3 years ago

It's been suggested that maybe mongodb is the source of the issue....I will try Redis soon.

Also, no "..." double quote lines in your alerta.conf was an issue I had to learn the hard way...

decentral1se commented 3 years ago

I will try Redis soon.

Ok, it's not simple and I am not sure how to do it since it now complains about a missing redis back-end and I see there is no documentation to show it is supported? Docs state that mongodb is supported but it is somehow problematic here for reasons unknown? I am sure that everything is working on my end now in terms of configuration and settings (see https://git.autonomic.zone/coop-cloud/alerta, docker swarm setup) and I am wondering how to go about investigating if Mongodb really is the issue?

decentral1se commented 3 years ago

OK fuck it, I couldn't resist more pain, using redis worked...

satterly commented 3 years ago

Kombu is the AMQP python package used to implement the mailer queue and it supports a number of different "transports" (aka. backends). See https://docs.celeryproject.org/projects/kombu/en/stable/introduction.html#transport-comparison

decentral1se commented 3 years ago

Well the MongoDB transport is broken for the mailer queue as far as I can see.

jrevillard commented 3 years ago

Dear @decentral1se, @satterly

I tried also but cannot make it working even with redis. I see that messages are published (amqp plugin logs)... The mailer is connected also... but no mail sent...

I don't know how to debug it....

Best.

mhk001 commented 3 years ago

I tried Mongo transport for mailer, 2 weeks ago and is working. OS ubuntu.bionic64 Docker environment, mongodb localy installed in web container DB postgresql

ENV variable ALERTA_CONF_FILE=/app/alerta.conf ALERTA_SVR_CONF_FILE=/app/alertad.conf

part of the alertad.conf: there is TOPIC notify with ' .... PLUGINS = ['debug','reject','blackout','enhance','normalise','acked_by','amqp'] AMQP_URL = 'mongodb://localhost:27017/kombu' AMQP_TOPIC = 'notify'

part of the alerta.conf there is amqp_topic notify without ' ... [alerta-mailer] key = some_string ; must have "write:heartbeats" scope mail_to = mail_from = from@domain.com amqp_url = mongodb://localhost:27017/kombu amqp_topic = notify dashboard_url = http://localhost:48080 smtp_host = smtp_port = smtp_username = smtp_login_name smtp_password = smtp_login_pass smtp_starttls = True smtp_use_ssl = False debug = True skip_mta = False email_type = html

alerta-mailer rules dir /app/alerta.rules.d

simple rule file: t1_rules.conf

[ { "name": "T001", "fields": [ {"field": "resource", "regex": "Mailer_D11" } ], "contacts": ["user@example.net"] } ]

Start alerta-mailer in debug mode #alerta-mailer

then send event from different shell #alerta send -r Mailer_D11 -E Test -e MailerErr_33 -S postfix4 -s critical

and alert mailer try deliver message after 30 sec and writes debug log on console

--- debug log --- DEBUG:mailer:Looking for rules files in /app/alerta.rules.d DEBUG:mailer:Parsing t1_rules.conf INFO:mailer:Adding rule {'name': 'T001', 'fields': [{'field': 'resource', 'regex': 'Mailer_D11'}], 'contacts': ['user@example.net']} to list of rules to be evaluated INFO:kombu.mixins:Connected to mongodb://localhost:27017/kombu Connected to mongodb://localhost:27017/kombu DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8080 Starting new HTTP connection (1): localhost:8080 DEBUG:urllib3.connectionpool:http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 297 http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 297 DEBUG:urllib3.connectionpool:http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 298 http://localhost:8080 "POST /api/heartbeat HTTP/1.1" 201 298 DEBUG:mailer:Initial contact list: [] Initial contact list: [] DEBUG:mailer:Checking 1 group rules Checking 1 group rules INFO:mailer:Evaluating rule T001 Evaluating rule T001 DEBUG:mailer:Evaluating rule field {'field': 'resource', 'regex': 'Mailer_D11'} Evaluating rule field {'field': 'resource', 'regex': 'Mailer_D11'} DEBUG:mailer:Trying to match Mailer_D11 to Mailer_D11 Trying to match Mailer_D11 to Mailer_D11 DEBUG:mailer:Extending contact to include ['user@example.net'] Extending contact to include ['user@example.net'] send: 'ehlo [10.8.0.2]\r\n' reply: b'250-XXXXXXXXX\r\n'

.......... SMTP conversation ....................

reply: retcode (235); Msg: b'2.7.0 Authentication successful' send: 'mail FROM: size=3099\r\n' reply: b'250 2.1.0 Ok\r\n' reply: retcode (250); Msg: b'2.1.0 Ok' send: 'rcpt TO:user@example.net\r\n' reply: b'556 5.1.10 user@example.net: Recipient address rejected: Domain example.net does not accept mail (nullMX)\r\n' reply: retcode (556); Msg: b'5.1.10 user@example.net: Recipient address rejected: Domain example.net does not accept mail (nullMX)' send: 'rset\r\n' reply: b'250 2.0.0 Ok\r\n' reply: retcode (250); Msg: b'2.0.0 Ok' ---------------------

Read this https://github.com/alerta/alerta-contrib/tree/master/integrations/mailer , Alerts have to meet the some criterias

BR.

jrevillard commented 3 years ago

Hello,

I finally made it working also... the main problem is that I think that you are missing debug messages to know when a message is acknowledged for different reasons.

Best, Jerome

satterly commented 2 years ago

@jrevillard Thanks!