search

bb-Ricardo / icinga-slack-bot

use slack to manage alarms in Icinga2
MIT License
5 stars 2 forks source link

readme

icinga-slack-bot

use slack to manage alarms in Icinga2

It can be used to interact with Icinga2 from your Slack client. It uses the Icinga2 API to get Host/Service status details. Simple status filters can be used to narrow down the returned status list.

Requirements

Installation

Setting up icinga-slack-bot

Now you would be able to start the bot with .pyenv/bin/python3 icinga-bot.py

Most likely the start will fail as the config is not fully set up.

It is recommended to create your own config cp icinga-bot.ini.sample icinga-bot.ini

Change config options according your environment. After you entered the Slack tokens you should be able to start the bot.

Run as a service

sudo cp icinga-slack-bot.service /etc/systemd/system
sudo systemctl daemon-reload
sudo systemctl start icinga-slack-bot
sudo systemctl enable icinga-slack-bot

Run with Docker

git clone https://github.com/bb-Ricardo/icinga-slack-bot.git
cd icinga-slack-bot
docker build -t icinga-bot .

Copy the config from the example to icinga-bot.ini and edit the settings.

Now you should be able to run the image with following command

docker run -d -v /PATH/TO/icinga-bot.ini:/app/icinga-bot.ini --name bot icinga-bot

Icinga API permissions

This would be an Icinga Slack bot API user

# vim /etc/icinga2/conf.d/api-users.conf

object ApiUser "icinga-bot" {
  password = "icinga"

  permissions = [ "objects/query/*", "objects/modify/*", "status/query", "actions/*" ]
}

For further details check the Icinga2 API documentation

Get Slack Bot Token

You need to install a Classic Slack Bot app in order to use RTM.

  1. Just give it a name and select the workspace.
  2. Go to "App Home" and use the Add legacy Bot User button
  3. Add a display name and the bot username like "icinga2"
  4. Go to "OAuth & Permissions" and use the Install App to Workspace button
  5. Confirm this in the bottom by selecting Allow
  6. Use the displayed Bot User OAuth Access Token to add to your config

You can also use this icon to represent the bot in Slack properly.

Configuration

icinga-slack-bot comes with a default config file

Run the bot

usage: icinga-bot.py [-h] [-c icinga-bot.ini] [-l {DEBUG,INFO,WARNING,ERROR}]
                     [-d]

This is an Icinga2 Slack bot.

It can be used to interact with Icinga2 from your Slack client. It uses the
Icinga2 API to get Host/Service status details. Simple status filters can be
used to narrow down the returned status list.

Version: 1.0.0 (2020-05-17)

optional arguments:
  -h, --help            show this help message and exit
  -c icinga-bot.ini, --config icinga-bot.ini
                        points to the config file to read config data from
                        which is not installed under the default path
                        './icinga-bot.ini'
  -l {DEBUG,INFO,WARNING,ERROR}, --log_level {DEBUG,INFO,WARNING,ERROR}
                        set log level (overrides config)
  -d, --daemon          define if the script is run as a systemd daemon

Use the bot

Following commands are currently implemented:

Help command

Each command also provides a detailed help help <command>

Command status filter

Following command filters are implemented

Command filter can be combined like "warn crit" which would return all services in WARNING and CRITICAL state. Also "problems" could be used to query all service with a certain name match in NOT OK sate.

Important:

Command name filter

You can add host names or services names to any status command. Also just parts of host and service names can be used to search for objects

Important:

HINT: You can also use quotes in your filter to find services containing spaces like:
ss server "network interfaces"

Using previous filter

Sometime you try to tackle a problem and get tired of entering the filter every time. In this case you can use !! which defaults to last commands filter.

Example:

You are checking the service status with ss myserver ntp. And then you want to reschedule this service. Instead of adding the filter again you can write rs !! which translates into rs myserver ntp. This works for all commands using filters.

Actions

Actions have been added to perform certain actions on hosts or services. Current actions are Acknowledgements and Downtimes.

Acknowledgements

This command will start a dialog to set an acknowledgement for an unhandled service or host. This can be started with this command and the bot will ask questions about the details on following order:

  1. host/service filter
  2. time when acknowledgement should expire (or never)
  3. a comment which should be added to the acknowledgement
INFO: time can be submitted in a relative format like:

tomorrow 3pm, friday noon or monday morning
Or more specific like january 2nd or even more specific like 29.02.2020 13:00. Just try and see what works best for you.
At the end the bot will ask you for a confirmation which can be answered with yes or just y or no. After that the bot will report if the action was successful or not.

SORT CUT:

It's also possible to short cut the whole Q/A and just issue the action in one command:

ack myserver ntp until tomorrow evening Wrong ntp config, needs update

This will acknowledge a problematic service ntp on myserver until 6pm the following day.

STRUCTURE:
ack <host> <service> until <time> <comment>

or

ack <host> until <time> <comment>

or

ack <service> until <time> <comment>

Downtimes

This command works pretty similar to the acknowledgement command except that the bot will ask for a downtime start. Here it's also possible to use a relative time format.

SORT CUT:

It's also possible to short cut the whole Q/A and just issue the action in one command:

dt myserver ntp from now until tomorrow evening NTP update

This will set a downtime for the service ntp on myserver until 6pm the following day.

STRUCTURE:
dt <host> <service> from <time> until <time> <comment>

or

dt <host> from <time> until <time> <comment>

or

dt <service> from <time> until <time> <comment>

Command examples

Important:

Help

help example

Help for command enable

help enable

Detailed host status example

detailed host answer

Detailed service status example

detailed service answer

Service name filter examples

detailed service answer detailed service answer

All problematic services

detailed service answer

Status overview

status overview answer

Acknowledge service problem

ack_service_problem

Disable notifications global

disable_notifications_global

Icinga status

icinga_status

Startup messages

Alert notification

To get Slack notifications if something goes wrong you can check out the notification handlers in contrib.

Copy slack-notification.sh to /etc/icinga2/scripts/ and add the icinga2_slack_notification_commands.conf to your icinga configuration (depends on your where you need to put this)

More information here: Icinga2 Docs -> notifications.

Don't forget to set NOTIFICATION_CONFIG in icinga2_slack_notification_commands.conf to the full path to your bot config file.

Slack WEBHOOK URL

In order to send alarms to Slack you need a Webhook URL.

  1. In your Slack Bot settings use Incoming Webhooks under Features.
  2. Switch Activate Incoming Webhooks to On
  3. Use Add New Webhook to Workspace
  4. Select the channel the alert messages should be posted to.
  5. Now you should have a Webhook URL you need to copy to your configuration

Alert examples

alert host down alert host up alert service problem

License

You can check out the full license here

This project is licensed under the terms of the MIT license.

Saying Thank You

quite some inspiration came from mlabouardy and his Go implementation of a slack bot