Set up logging and set up monitoring

[urbanslug]

Karma needs to have a way of getting logging done that works.

Have various log levels.

Should work well with Docker.

Implementation plan

• [X] Use a simple logging lib such as https://pimterry.github.io/loglevel/
• [X] Save a logs in a log with karma-error.log karma-warn.log karma-info.log

  Can be saved in e.g /var/.../karma/karma-[a-z]{0,3}.log. Since it won't be running in debug mode no need for a karma-debug log.

• [X] Write it such that it can take advantage of https://github.com/Unitech/pm2
• [X] Version control the vars other deployment deps information into the playbooks
• [X] Add a karma.md in the playbooks/docs folder
• [X] Hook it up to uptime robot

[pld]

Cool, what events or categories of events do we want to log here?

[urbanslug]

@pld Info, warn and error. What do you think?

[urbanslug]

Thinking we can also log fatal?

[ukanga]

I would think uptime robot is a monitoring tool rather than a logging tool. Not sure how it relates to this.

[ukanga]

What to consider sometimes depends on where you want the logs to be sent. Is it being sent to a log file, to the operating system event logger or an app like sentry? You could get a logger that logs to a file and use Logstash to send it to elastic search or Graylog like we do Nginx access logs.

[urbanslug]

I've renamed the issue to implement logging and set up monitoring.

What I want is to make sure I know when something happens and have it restarted when it crashes.

Not excited about relying on a proprietary service such as sentry for this but given it's using Facebook Messenger we can go on and use it.

[pld]

What sort of logs are you talking about here, and what sort of value are you planning to get from them?

On Fri, Jul 21, 2017 at 03:58 ukanga notifications@github.com wrote:

What to consider sometimes depends on where you want the logs to be sent. Is it being sent to a log file, to operating system event logger or an app like sentry? You could get a logger that logs to a file and use logstash to send it to elastic search or graylog like we do nginx access logs.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/onaio/karma/issues/14#issuecomment-316932630, or mute the thread https://github.com/notifications/unsubscribe-auth/AADGEhIu6Q4yczWCnoTENtOuB3ZOqkufks5sQFoWgaJpZM4OcbHy .

[pld]

Sentry is open source, I recommend it.

For making sure the service is running, that's something monit and well written deploy scripts are for.

On Fri, Jul 21, 2017 at 04:04 Njagi Mwaniki notifications@github.com wrote:

I've renamed the issue to implement logging and set up monitoring. Generally what I want is to make sure I know when something happens and have it restarted when it crashes.

Not excited about relying on a proprietary service such as sentry for this but given it's using Facebook Messenger I guess we can go on and use it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/onaio/karma/issues/14#issuecomment-316933897, or mute the thread https://github.com/notifications/unsubscribe-auth/AADGEhyApv487n-rmXAfZZgFPUD3m3s5ks5sQFuBgaJpZM4OcbHy .

[ukanga]

As such I think, logging is a very separate issue from monitoring. For logging, this is primarily sent to stdout or a log file. One thing you can determine is what category of error do you need to be notified? In onaio/onadata we get emails for all 500 exceptions, and certain errors we explicitly send emails using the mail_admins() function. For monitoring, we use monit to be notified when the service is up, using too much memory, running out of storage among other things. You can use certain apps which are receiving your logs to set alarms.

You just need to identify what exactly it is you want to do. Otherwise this topic is overly broad.

Winston and bunyan seem popular, you should consider those two.

[urbanslug]

Yes monitoring is a separate issue but I want to do it and handle/track it all in one place (the github issue tracker) since I can't think of a better place to handle it/have a conversation about it. Yes, the topic is overly broad so I would like to get other's opinions before I do something that is totally off.

I want to log:

log level	among potential causes
info	sent this, received this
debug	initialized conversationX, listening on
warning	many restarts, null conversations, unexpected conversation flow
errors	couldn't parse/handle user input, unexpected conversation restart
fatal	lack of an env var needed to start the app, > 0 exit codes and reasons

The value I want to get out of them is:

• know when something happens
• what caused it
• have enough info to stop it from reoccurring.

[urbanslug]

Botkit has logging already done for us and seems it can be hooked up to existing tools like winston https://github.com/howdyai/botkit/blob/master/docs/logging.md