Alerting based on CoreOS Kernel events

[sukrit007]

Currently, when the container exceeds the memory limit set by docker args, the kernel would kill the process however no notification is sent when such an event happens.

We need some strategy to notify on OOM Kill events. The event can be detected from systemd journal logs for kernel. " Eg: Memory cgroup out of memory: Kill process 7232 (java) score 1014 or sacrifice child"

[sukrit007]

Options:

I can think of 3 options:

• Adding grok based matcher to syslog config and email output : http://drewsearcy.com/email-alerts-with-logstash
• Use Watcher provided by ELK stack https://www.elastic.co/guide/en/watcher/current/introduction.html
• Modify log service to add support for monitoring events matching certain criteria and do alerting.

I am kind of inclined towards option 2 as it can easily be managed and configured using RESTFul API. The only downside is that it requires plugin installation.

[sukrit007]

Currently we decided to use logstash hipchat plugin :

if [syslog_program] == 'kernel' and [short_message] =~ ".*Memory cgroup out of memory.*" {
    hipchat {
      room_id => "<room>"
      token => "<token>"
      color => "red"
      trigger_notify => true
    } 
  }