Logging / Error handling

[bjacobel]

• [x] Implement Sentry
• [x] Get the following into rsyslog:
  • [x] redis
  • [x] memcached
  • [x] rq / rq-scheduler
  • [x] elasticsearch
  • [x] nginx
  • [x] gunicorn
  • [x] postgres
  • [x] mysql
  • [x] fail2ban
  • [x] ufw
  • [x] new relic
• [x] get rsyslog into Logentries or other log collection service
• [ ] Customize deploy/maintenance (503) page

[bjacobel]

Been working on this for a week or so, I've decided to go with a hosted ELK stack and something local and small to send the logs there (terminology for this appears to be "log shipper", but there's tons of unclear terminology in this space? also lumberjack puns...). Tried Beaver and it's been iffy to configure and appears to use a huge amount of CPU. It's also Python2-only, I gave a crack at updating it but it has turtles all the way down in unmaintained py2x-only dependencies.

I'm looking into logstash-forwarder which is golang (+1) and from Elastic themselves (+1) and configured in json (+1) but appears to not be actively maintained (-1) and it's not actually clear if it can integrate with Logsene without an intermediary (-1).

EDIT: logstash-forwarder integrates directly with Logz.io which has a free plan (+1) with 5 day retention (meh) and better UI than Logsene (+1)) so I'm thinking that's the direction this should go in.