habitat daemons should mail us when there's an unhandled exception

[danielrichman]

see habitat/utils/immortal_changes.py

[danielrichman]

See first code snippet in http://flask.pocoo.org/docs/errorhandling/ : add a SMTPHandler to the standard python logging object; job done. Errors now get mailed to you.

Trivially added to config and startup.setup_logging

[danielrichman]

• don't forget: http://flask.pocoo.org/docs/errorhandling/#email pretty formatting

[danielrichman]

also good would be a supervisord event listener ( http://supervisord.org/events.html#example-event-listener-implementation ) that emails us if a daemon stops, fails to start, e.t.c.

[danielrichman]

This is done for the parser and spacenearus daemons (pending deployment) but is still TODO for calendar, transition, maybe couch named python and the supervisord event listener thing.

[danielrichman]

dug through the code and found this snippet in ~habitat/old/habitat-calendar/blah/cal.py; saving it here in case it's lost in spring cleaning or something

+ADMINS = ['daniel@kraken.habhub.org', 'adam@kraken.habhub.org']
+if not app.debug:
+    import logging
+    from logging.handlers import SMTPHandler
+    mail_handler = SMTPHandler('127.0.0.1', 'habitat@kraken.habhub.org', ADMINS
+                               'calendar error')
+    mail_handler.setLevel(logging.ERROR)
+    app.logger.addHandler(mail_handler)
+
+    @app.route("/test_error")
+    def test_error():
+        raise AssertionError("This is a test error message")

Though we should probably import and use habitat.setup_logging instead somehow with the global habitat.yml file (yay).

[danielrichman]

http://pypi.python.org/pypi/superlance provides 'crashmailbatch' and 'failmailbatch' which do pretty much exactly what we want for a supervisord event listener.

[danielrichman]

superlance deployed. Config was:

[eventlistener:crashmailbatch]
user=nobody
command=/opt/superlance/bin/crashmailbatch -t root@kraken.habhub.org -f supervisord@kraken.habhub.org
events=PROCESS_STATE,TICK_60

[eventlistener:fatalmailbatch]
user=nobody
command=/opt/superlance/bin/fatalmailbatch -t root@kraken.habhub.org -f supervisord@kraken.habhub.org
events=PROCESS_STATE,TICK_60

[adamgreig]

:cool: :+1:

[danielrichman]

List of daemons/things that might need to email us

Done:

• spacenearus uploader
• parser
• supervisord
• transition (wsgi)
• calendar

Not done:

• couch view server

[danielrichman]

18:39 < DanielRichman> you think emails from the parser are bad
18:39 < DanielRichman> if the view server is sad and can email you about it
18:39 < DanielRichman> you're gonna know

Couch view server being able to email us would be suicide.

Maybe we want to investigate one of:

• getsentry/airbrake/hoptoad/exceptional/errbit
• logging handler that throttles itself
• log serious stuff to a 'paniclog' (like exim) which is checked by a cronjob or monit.