search

nisabek / icinga2-slack-notifications

Icinga2 notification integration with slack
Apache License 2.0
71 stars 36 forks source link

readme

icinga2-slack-notifications

Icinga2 notification integration with slack

Overview

Native, easy to use Icinga2 NotificationCommand to send Host and Service notifications to pre-configured Slack channel - with only 1 external dependency: curl

Also available on Icinga2 exchange portal

What will I get?

Why another approach?

We found the following 2 existing Icinga2 to Slack integrations.

Installation

Installation using Debian package

!NOTE: At this moment debian package is behind the master code. If you want to use this plugin with Icinga Director, please make sure to install from Git (see below) We're working on updating the repo in the meantime.

We use reprepro to distribute our package from github. You would need to install apt-transport-https that supports adding an https based repository to the debian repo list.

here are the steps to perform:

foo@bar:~# apt-get install -y apt-transport-https
foo@bar:~# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 10779AB4
foo@bar:~# add-apt-repository "deb https://raw.githubusercontent.com/nisabek/icinga2-slack-notifications/master/reprepro general main"
foo@bar:~# apt-get update

You are now ready to install the plugin with 

foo@bar:~# apt-get install icinga2-slack-notifications

This will create the plugin files in the correct icinga2 conf directory.

Installation using git

  1. Clone the repository and copy the relevant folder into your Icinga2 /etc/icinga2/conf.d directory
foo@bar:~# git clone git@github.com:nisabek/icinga2-slack-notifications.git /opt/icinga2-slack-notifications
foo@bar:~# cp -r /opt/icinga2-slack-notifications/src/slack-notifications /etc/icinga2/conf.d/
  1. Use the slack-notifications-user-configuration.conf.template file as reference to configure your Slack Webhook URL and Icinga2 Base URL to create your own slack-notifications-user-configuration.conf
foo@bar:~# cp /etc/icinga2/conf.d/slack-notifications/slack-notifications-user-configuration.conf.template /etc/icinga2/conf.d/slack-notifications/slack-notifications-user-configuration.conf
  1. Fix permissions
foo@bar:~# chown -R root:nagios /etc/icinga2/conf.d/slack-notifications
foo@bar:~# chmod 0750 /etc/icinga2/conf.d/slack-notifications
foo@bar:~# chmod 0640 /etc/icinga2/conf.d/slack-notifications/*

Configuration

Icinga2 features

In order for the slack-notifications to work you need at least the checker, command and notification icinga2 features enabled.

In order to see the list of currently enabled features execute the following command

foo@bar:~# icinga2 feature list

In order to enable a feature use 

foo@bar:~# icinga2 feature enable FEATURE_NAME

Notification configuration

  1. Configure Slack Webhook and Icinga2 web URLs in /etc/icinga2/conf.d/slack-notifications/slack-notifications-user-configuration.conf

    template Notification "slack-notifications-user-configuration" {
import "slack-notifications-default-configuration"

vars.slack_notifications_webhook_url = "<YOUR SLACK WEBHOOK URL>, e.g. https://hooks.slack.com/services/TOKEN1/TOKEN2"
vars.slack_notifications_icinga2_base_url = "<YOUR ICINGA2 BASE URL>, e.g. http://icinga-web.yourcompany.com/icingaweb2"
}
...

  2. In order to enable the slack-notifications for Services add vars.slack_notifications = "enabled" to your Service template, e.g. in /etc/icinga2/conf.d/templates.conf

 template Service "generic-service" {
   max_check_attempts = 5
   check_interval = 1m
   retry_interval = 30s

   vars.slack_notifications = "enabled"
 }

In order to enable the slack-notifications for Hosts add vars.slack_notifications = "enabled" to your Host template, e.g. in /etc/icinga2/conf.d/templates.conf

 template Host "generic-host" {
   max_check_attempts = 5
   check_interval = 1m
   retry_interval = 30s

   vars.slack_notifications = "enabled"
 }

Make sure to restart icinga after the changes

foo@bar:~# systemctl restart icinga2
  1. Further customizations [optional]

You can customize the following parameters of slack-notifications :

In order to do so, place the desired parameter into slack-notifications-user-configuration.conf file.

Note

Objects as well as templates themselves can import an arbitrary number of other templates. Attributes inherited from a template can be overridden in the object if necessary.

The slack-notifications-user-configuration section applies to both Host and Service, whereas the slack-notifications-user-configuration-hosts and slack-notifications-user-configuration-services sections apply to Host and Service respectively

Example channel name configuration for Service notifications

template Notification "slack-notifications-user-configuration" {
    import "slack-notifications-default-configuration"

    vars.slack_notifications_webhook_url = "<YOUR SLACK WEBHOOK URL>, e.g. https://hooks.slack.com/services/TOKEN1/TOKEN2"
    vars.slack_notifications_icinga2_base_url = "<YOUR ICINGA2 BASE URL>, e.g. http://icinga-web.yourcompany.com/icingaweb2"
}

template Notification "slack-notifications-user-configuration-hosts" {
    import "slack-notifications-default-configuration-hosts"

    interval = 1m
}

template Notification "slack-notifications-user-configuration-services" {
    import "slack-notifications-default-configuration-services"

    interval = 3m

    vars.slack_notifications_channel = "#monitoring_alerts_for_service"
}

You can choose to override the whole icon dictionary, or override specific types only:

Example override the whole icon dictionary

template Notification "slack-notifications-user-configuration" {
    import "slack-notifications-default-configuration"

    vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/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"
    vars.slack_notifications_icinga2_base_url = "http://localhost:80/icingaweb2"
    vars.slack_notifications_channel = "#icinga2-private-test"
    vars.slack_notifications_icon_dictionary = {
       "DOWNTIMEREMOVED" = "leftwards_arrow_with_hook",
       "ACKNOWLEDGEMENT" = "ballot_box_with_check",
       "PROBLEM" = "bomb",
       "RECOVERY" = "large_blue_circle",
       "DOWNTIMESTART" = "up",
       "DOWNTIMEEND" = "arrow_double_down",
       "FLAPPINGSTART" = "small_red_triangle",
       "FLAPPINGEND" = "small_red_triangle_down",
       "CUSTOM" = "speaking_head_in_silhouette"
    }    
    ...

Example override specific type

template Notification "slack-notifications-user-configuration" {
    import "slack-notifications-default-configuration"

    vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/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"
    vars.slack_notifications_icinga2_base_url = "http://localhost:80/icingaweb2"
    vars.slack_notifications_channel = "#icinga2-private-test"

    vars.slack_notifications_icon_dictionary.CUSTOM = "cherries"
    ...
}

If you, for some reason, want to disable the slack-notifications from icinga2 change the following parameter inside the corresponding Host or Service configuration object/template:

vars.slack_notifications == "disabled"

Besides configuring the slack-notifications parameters you can also configure other Icinga2 specific configuration parameters of the Host and Service, e.g.:

How it works

slack-notifications uses the icinga2 native [NotificationCommand] (https://docs.icinga.com/icinga2/latest/doc/module/icinga2/chapter/object-types#objecttype-notificationcommand) to collect the required data and send a message to configured slack channel using curl

The implementation can be found in slack-notifications-command.conf and it uses Lambdas!

Testing

Since the official docker image of icinga2 seems not to be maintained, we've been using jordan's icinga2 image to test the notifications manually.

Usual procedure for us to test the plugin is to 

foo@bar:~# docker run -p 8081:80 --name slack-enabled-icinga2 -v $PWD/icinga2-docker-volume:/etc/icinga2 -idt jordan/icinga2:latest
foo@bar:~# docker cp src/templates.conf slack-enabled-icinga2:/etc/icinga2/conf.d/
foo@bar:~# docker cp src/slack-notifications slack-enabled-icinga2:/etc/icinga2/conf.d/
foo@bar:~# docker restart slack-enabled-icinga2

after that navigate to http://localhost:8081/icingaweb2 and try out some notifications.

We understand that this is far from automated testing, and we will be happy to any contributions that would improve the procedure.

Troubleshooting

The slack-notifications command provides detailed debug logs. In order to see them, make sure the debuglog feature of icinga2 is enabled.

foo@bar:~# icinga2 feature enable debuglog

After that you should see the logs in /var/log/icinga2/debug.log file. All the slack-notifications specific logs are pre-pended with "debug/slack-notifications"

Use the following grep for troubleshooting: 

foo@bar:~# grep "warning/PluginNotificationTask\|slack-notifications" /var/log/icinga2/debug.log
foo@bar:~# tail -f /var/log/icinga2/debug.log | grep "warning/PluginNotificationTask\|slack-notifications"

Useful links

Running with Icinga Director

There has been some discussion over here on how to run the plugin with Icinga Director. We'd appreciate somebody going over this part of documentation and verifying it.

Main points to make it work:

Latest how-to on Icinga Director

Thanks a lot to @jwhitbread for putting down these steps in the issue