how to configure with icinga-director

[lcorsini]

I've tried installing this component, but I'm stuck, since I'm using icinga-director. I've installed and configured the deb package. In icinga-director added the vars.slack_notifications to my hosts, but when I apply the config it says critical/config: Error: Import references unknown template: 'slack-notifications-user-configuration-hosts' Location: in /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf: 47:3-47:55 /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(45): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(46): apply Notification "slack-notifications-notification-hosts" to Host { /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(47): import "slack-notifications-user-configuration-hosts" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ because director doesn't know anything about this new template

[staenker]

I'll get to it this Friday, cheers, Richard

[lcorsini]

Hi, any update on this?

[majales]

Hi.. I did it on Director. I created Data Field slack_notifications and added it to host-template in Director and set it to "enabled" .. and added it to service-template in Director also with "enabled" .. It did the trick.. I had not modified hosts and service template file,s it is not necessary.. It was on icinga 2.7 and director 1.3.1 ( slightly newer then release.. It was taken from github repo in April )

[rgevaert]

Same here!

[flaviomeyer]

Hi @nisabek

Thanks for this nice notification plugin. I'm looking Icinga Director install guide. Is it possible to create a guide for Icinga Director?

@majales Maybe you can post your solution?

Thanks :)

Flavio

[nisabek]

Hi all,

Thanks for addressing this, and sorry for late response. I just got to try icinga-director, and in order for the director to recognize the changes in the icinga core I run the command icingacli director kickstart run

That should be run after the usual installation and final restart of the icinga2 service. After that the slack-notifications-command is visible in the command list of the icinga director, and if your hosts have the vars.slack_notifications = "enabled" set it will fire the notifications.

@lcorsini @meyerf99 - I would appreciate if you give it a try and report feedback.

Thanks and Regards, Nune

[flaviomeyer]

Hey @nisabek

Thanks for your fast answer. I got this error with timeperiod.

critical/config: Error: Validation failed for object 'x.ch!disk_check!slack-notifications-notification-services' of type 'Notification'; Attribute 'period': Object '24x7' of type 'TimePeriod' does not exist. Location: in /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf: 8:5-8:19 /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(6): user_groups = [ "icingaadmins" ] /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(7): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(8): period = "24x7" ^^^^^^^^^^^^^^^ /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(9): /etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf(10): interval = 5m Maybe you know how i can fix it?

Kind regards, Flavio

[rgevaert]

It's possible to manage the custom attributes on host/service level through director. However configuring the other things isn't possible. Or at least I didn't find how. I added them to the configuration files.

[flaviomeyer]

I've fixed my problem, but it was very hard. @nisabek Maybe u can documentation with timeperiod and icingaadmins group as prerequisite. Is it possible to test notifications?

From my point of view, I have everything set up, but unfortunately I think no message in Slack. Does the script have a logfile? If yes, where?

Kind regards,

Flavio

PS: In my Debug Log i see only this: [2018-03-01 20:58:04 +0100] debug/slack-notifications: Sending notification...read user configuration successfully: {"vars.slack_notifications_botname":null,"vars.slack_notifications_channel":null,"vars.slack_notifications_icinga2_base_url":null,"vars.slack_notifications_icon_dictionary":null,"vars.slack_notifications_plugin_output_max_length":0.0,"vars.slack_notifications_webhook_url":null}

I think thats false, coz all infos have null as string, is that right?

[lcorsini]

In the meantime, I've actually reinstalled my infrastructure without director so I cannot really test this anymore in the short term.

[flaviomeyer]

@nisabek I've fixed my problem.

Set in Icinga Director Notification Interval to 60 and not 0.

See this link in: https://monitoring-portal.org/woltlab/index.php?thread/41134-slack-notification-im-director-per-user-zuweisung/

[nisabek]

seems like this is fixed. The current master contains the reference to a docker image that can be used for testing.

[nisabek]

It looks like it's still unclear how to configure the plugin in Icinga Director. @majales @rgevaert - could you clarify if you had any luck configuring the webhook_url and making it work in Icinga Director? If so, I'd be very thankful for a README contribution with steps how to achieve that.

Thanks! Nune

[rgevaert]

Sorry, I'm not using icinga2 anymore (different job) so can't help unfortunately.

[buggsdummy]

Can someone update this? Thanks!

[flaviomeyer]

Interesting... I have looked for this notification addon and found out that is not working anymore. Maybe I build in the next few days a new slack notification plugin which works with Icinga Director.

[NC-Yungd]

After running the kickstart wizard in Icinga2 Director, is this the expected output for the NotificationCommand?

I am unable to get any messages in my Slack channel. Below are from my debug logs:

# tail -f /var/log/icinga2/debug.log | grep "warning/PluginNotificationTask\|slack-notifications"
[2019-12-12 17:45:15 +0000] notice/Notification: Attempting to send notifications of type 'Custom' for notification object 'server1.com!slack-notifications-notification-hosts'.
[2019-12-12 17:48:22 +0000] notice/NotificationComponent: Attempting to send reminder notification 'server2.com!linux_apt!slack-notifications-notification-services'.
[2019-12-12 17:48:22 +0000] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'server3.com!linux_apt!slack-notifications-notification-services'.
[2019-12-12 17:48:22 +0000] notice/NotificationComponent: Attempting to send reminder notification 'server4.com!linux_apt!slack-notifications-notification-services'.

[nisabek]

@NC-Yungd can't verify right now, if that's the expected output, but I can tell you that those logs are not from the plugin(your grep is correct, just it matches some other lines...)...are you sure everything is enabled on the plugin side?

[flaviomeyer]

@NC-Yungd A few weeks ago I have rebuilt the Slack notification. If I find time tomorrow I'il create a new gist and I'il let you know.

[NC-Yungd]

Here are my current configurations:

/etc/icinga2/zones.conf

object Endpoint "icinga2.xyz.com" {
}

object Zone "master" {
    endpoints = [ "icinga2.xyz.com" ]
}

object Zone "global-templates" {
    global = true
}

object Zone "director-global" {
    global = true
}

/etc/icinga2/zones.d# tree -d

.
├── global-templates
│   ├── commands
│   └── templates
└── master
    ├── archive
    └── slack-notifications

/etc/icinga2/zones.d/master/slack-notifications# cat slack-notifications-user-configuration.conf

template Notification "slack-notifications-user-configuration" {
    import "slack-notifications-default-configuration"

    vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/secret1/secret2"
    vars.slack_notifications_icinga2_base_url = "http://icinga2.xyz.com"
    vars.slack_notifications_channel = "#icinga2_alerts"
    vars.slack_notifications_botname = "icinga2_bot"
}

template Notification "slack-notifications-user-configuration-hosts" {
    import "slack-notifications-default-configuration-hosts"
}

template Notification "slack-notifications-user-configuration-services" {
    import "slack-notifications-default-configuration-services"
}

In Icinga2 Director, I've created the custom variable for slack_notifications as a string. Afterwards, I've add ed slack_notifications as a field to my Host and Services template and included the string "enabled".

[nisabek]

hmm, from the fact that there are no logs of the plugin, I suppose that the slack_notifications variable is not understood by the director... when you say "custom variable for slack_notifications as a string" do you mean a data field? Cause @majales mentioned creating a _datafield and I'm not sure if that's the same as "custom variable"

Sorry I can't do more at the moment :/

[NC-Yungd]

@nisabek Initially, I installed this plugin by using the "Debian package" method, but after installing with git and copying the slack-notification conf files into my Icinga2 zones.d/master folder, I was able to get notifications in Slack.

I'm going to assume that the Debian package is out-of-date since I wasn't able to get Slack notifications with Director.

[nisabek]

Thanks @NC-Yungd for getting back to us and making it work! I'm having a bit of a trouble updating the dbian repo, in the meantime I updated the README

https://github.com/nisabek/icinga2-slack-notifications#installation-using-debian-package https://github.com/nisabek/icinga2-slack-notifications#running-with-icinga-director

If you feel like there's information missing regarding making it work on icinga director, I'd appreciate a a PR with more details.

Thanks! Nune

Edit: updated wrong links.

[guzaho]

Hi there!

Thanks for creating such a useful plugin.

I recently installed icinga2 with icingaweb2 and director, and I need to integrate is to my company's slack.

I've followed the howto and installed from git (and discovered a few mistakes in the howto), as suggested, but I cannot make it work.

This is what I have done:

ONE) Created a slack webhook and verified with curl that I do indeed get messages into the right channel.

TWO) Logged into the icinga2 master machine as root, and

cd /etc/icinga2/conf.d/ git clone https://github.com/nisabek/icinga2-slack-notifications.git cd icinga2-slack-notifications cp src/slack-notifications/slack-notifications-user-configuration.conf.template slack-notifications-user-configuration.conf chown -R nagios:nagios /etc/icinga2/conf.d

THREE) Customized /etc/icinga2/conf.d/templates.conf

template Host "generic-host" { max_check_attempts = 3 check_interval = 1m retry_interval = 30s

check_command = "hostalive" vars.slack_notifications = "enabled" }

/** * Provides default settings for services. By convention * all services should import this template. */ template Service "generic-service" { max_check_attempts = 5 check_interval = 1m retry_interval = 30s

vars.slack_notifications = "enabled" }

FOUR) Customized /etc/icinga2/conf.d/icinga2-slack-notifications/slack-notifications-user-configuration.conf

template Notification "slack-notifications-user-configuration" { import "slack-notifications-default-configuration"

vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/secret1/secret2" vars.slack_notifications_icinga2_base_url = "https://my.company.com/icingaweb2" vars.slack_notifications_channel = "#service-health" vars.slack_notifications_botname = "service_health_bot" }

template Notification "slack-notifications-user-configuration-hosts" { import "slack-notifications-default-configuration-hosts" }

template Notification "slack-notifications-user-configuration-services" { import "slack-notifications-default-configuration-services" }

FIVE) Enable Icinga2 debug logging icinga2 feature enable debuglog

SIX) Restarted icinga2 systemctl restart icinga2

SEVEN) Triggered an alert by stopping a service, and verified that icingaweb2 shows the alert

But I don't see any slack in the debug file. grep "warning/PluginNotificationTask\|slack-notifications" /var/log/icinga2/debug.log

I have not configured anything in the director, because I'm new to Icinga2, and the discussions are not clear on what to do in what order.

I don't have zones configured, do I need to?

Can anyone please help?

[nisabek]

hi @guzaho and thanks for using the plugin It's getting hard to solve these issues, since everyone's setup looks different, but I'm gonna try.

question - does the service you stopped to test, use the generic-service template?... Can you show how it is configured?... another question - Do you see any logs in the debug.log?..

[guzaho]

Hi @nisabek !

Thanks for the prompt reply!

I do see plenty of logs in the debug.log.

Actually, right before I saw your message now, I got a message in slack! Yay!

It seems however, that I am only getting messages for the master node.

The service I stopped is checked by check_systemd, and my service template looks like this in director:

zones.d/director-global/service_templates.conf template Service "check_systemd" { check_command = "check_systemd" command_endpoint = host_name vars.slack_notification = "enabled" }

The service set that uses the service template is like this:

apply Service "check_systemd" { import "check_systemd"

assign where "sentinel-template" in host.templates

import DirectorOverrideTemplate

}

So I guess it does not use the generic-service template?

[nisabek]

oh wait, that's a template on it's own, but do I see a typo?... vars.slack_notifications = "enabled"

[guzaho]

ok, fixed the missing s

zones.d/director-global/service_templates.conf template Service "check_systemd" { check_command = "check_systemd" command_endpoint = host_name vars.slack_notifications = "enabled" }

[nisabek]

and?...works?.. :)

[guzaho]

No, not for other hosts.. just the master... which does only use default services like apt, procs, etc. My nodes only use check_apt and check_procs, etc.

The particular service I'm testing does not post any msgs to slack, it seems...

[guzaho]

I've followed the icinga2 official docs to the point. I've set up service templates, then service groups and the assigned that to hosts. So any service that inherits the template for check_systemd has now gotten the slack_notifications set to "enabled":

[nisabek]

alrighty that looks correct...questions

1. you say, you get notifications from your 'master' node - which template does the master node use?.. on a side note - you must be then seeing logs from the master node...
2. in the comment by @NC-Yungd it is mentioned that "In Icinga2 Director, I've created the custom variable for slack_notifications as a string. Afterwards, I've add ed slack_notifications as a field to my Host and Services template and included the string "enabled"."

can you try that?...

[guzaho]

Of course, I can try! Thank you!

1. Yes, I do see slack messages in the debug log now, but only for the master node.

I am not sure which template the master node uses, it just showed up after installing icinga2, icingaweb2 and the director.. How can I check?

2. Yes, I did as @NC-Yungd suggested. I've created the slack_notifications as a string in data fields:

I have now also set the slack_notifications to enabled for the host, and it works!!

[guzaho]

Ok, so just to sum up, the missing piece what that misspelling that check_notifications should be in plural form, and that the hosts did not have the slack_notifications set to "enabled"!

thank you so much!

[nisabek]

I looked through the docs again, and your configuration, and I don't seem to find if you have this piece somewhere

apply Notification "slack-notifications-notification-services" to Service {
  import "slack-notifications-user-configuration-services"

  assign where service.vars.slack_notifications == "enabled"
}

[guzaho]

Hm, that's true. Where does that go?

[nisabek]

ah, ignore that, that's in the source code! so you must have checked it out :D my memory of the project is not that fresh sorry :)

[nisabek]

so - works now?

[guzaho]

Yes, looks like it. I'm gonna do some more testing, just in case. But all good for now! Thank you so much. :smile:

[nisabek]

awesome, enjoy!

I really want to close this issue cause it's turned into debugging session of different issues.

Anybody having trouble with icinga2 director configuration please refer to last part of the docs https://github.com/nisabek/icinga2-slack-notifications#running-with-icinga-director

If that doesn't help - please open a new issue. And of course, contribute - with more docs, more tests, ...! ;)