Notification is not sent because the default notification script (/etc/icinga2/scripts/mail-service-notification.sh) is not working.
Possible Solution
Fix the default notification script or use an alternative.
Steps to Reproduce (for bugs)
Force a notification
Context
This is a new and empty Icinga2 installation. I've added a master and satellite setup and wanted to test notifications, when I came across the bug. With debug log activated:
[2018-12-08 12:46:08 +0000] notice/Notification: Attempting to send notifications for notification object 'bw-nas!Diskspace /srv/data!service-default'.
[2018-12-08 12:46:08 +0000] notice/Notification: Not checking notification filters for notification object 'bw-nas!Diskspace /srv/data!service-default': Notification was forced.
[2018-12-08 12:46:08 +0000] notice/Notification: Not checking notification filters for notification object 'bw-nas!Diskspace /srv/data!service-default' and user 'ckadm': Notification was forced.
[2018-12-08 12:46:08 +0000] information/Notification: Sending 'Custom' notification 'bw-nas!Diskspace /srv/data!service-default' for user 'ckadm'
[2018-12-08 12:46:08 +0000] debug/DbEvents: add notification history for 'bw-nas!Diskspace /srv/data'
[2018-12-08 12:46:08 +0000] debug/DbEvents: add contact notification history for service 'bw-nas!Diskspace /srv/data' and user 'ckadm'.
[2018-12-08 12:46:08 +0000] notice/ApiListener: Relaying 'event::SetForceNextNotification' message
[2018-12-08 12:46:08 +0000] notice/ApiListener: Relaying 'event::SendNotifications' message
[2018-12-08 12:46:08 +0000] notice/ApiListener: Relaying 'event::SetForceNextNotification' message
[2018-12-08 12:46:08 +0000] notice/ApiListener: Relaying 'event::NotificationSentToAllUsers' message
[2018-12-08 12:46:08 +0000] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh': PID 14812
[2018-12-08 12:46:08 +0000] debug/DbEvents: add log entry history for 'bw-nas!Diskspace /srv/data'
[2018-12-08 12:46:08 +0000] information/Notification: Completed sending 'Custom' notification 'bw-nas!Diskspace /srv/data!service-default' for checkable 'bw-nas!Diskspace /srv/data' and user 'ckadm'.
[2018-12-08 12:46:08 +0000] notice/ApiListener: Relaying 'event::NotificationSentUser' message
[2018-12-08 12:46:08 +0000] notice/Process: PID 14812 ('/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 1
[2018-12-08 12:46:08 +0000] warning/PluginNotificationTask: Notification command for object 'bw-nas!Diskspace /srv/data' (PID: 14812, arguments: '/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 1, output: Required parameter 'SERVICENAME' is missing.
This is the default notification script coming with the icinga2 package so I assume it should be working out of the box.
Your Environment
Version used (icinga2 --version):
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.2-1)
Copyright (c) 2012-2018 Icinga Development Team (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 3.16.0-4-amd64
Architecture: x86_64
Build information:
Compiler: GNU 6.3.0
Build host: 486e413fb159
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
* If you run multiple Icinga 2 instances, the `zones.conf` file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes.
Irrelevant
Ah crap. Sometimes I find the solution right after opening an issue.
Problem is in the definition of the "NotificationCommand" which I copied from another instance. Sorry!
Expected Behavior
Notification is sent.
Current Behavior
Notification is not sent because the default notification script (/etc/icinga2/scripts/mail-service-notification.sh) is not working.
Possible Solution
Fix the default notification script or use an alternative.
Steps to Reproduce (for bugs)
Context
This is a new and empty Icinga2 installation. I've added a master and satellite setup and wanted to test notifications, when I came across the bug. With debug log activated:
This is the default notification script coming with the icinga2 package so I assume it should be working out of the box.
Your Environment
icinga2 --version
):Copyright (c) 2012-2018 Icinga Development Team (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
System information: Platform: Debian GNU/Linux Platform version: 9 (stretch) Kernel: Linux Kernel version: 3.16.0-4-amd64 Architecture: x86_64
Build information: Compiler: GNU 6.3.0 Build host: 486e413fb159
Application information:
General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2
Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var
Internal paths: Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid
PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
Disabled features: compatlog debuglog elasticsearch gelf graphite livestatus opentsdb perfdata statusdata syslog Enabled features: api checker command ido-mysql influxdb mainlog notification3
[2018-12-08 12:55:13 +0000] information/cli: Icinga application loader (version: r2.10.2-1) [2018-12-08 12:55:13 +0000] information/cli: Loading configuration file(s). [2018-12-08 12:55:13 +0000] information/ConfigItem: Committing config item(s). [2018-12-08 12:55:13 +0000] information/ApiListener: My API identity: test.example.com [2018-12-08 12:55:13 +0000] warning/ApplyRule: Apply rule 'System Updates' (in /etc/icinga2/zones.d/global-templates/applyrules/updates.conf: 11:1-11:30) for type 'Service' does not match anywhere! [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 119 Services. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 InfluxdbWriter. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 IcingaApplication. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 10 Hosts. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 FileLogger. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 3 Dependencies. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 4 NotificationCommands. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 129 Notifications. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 NotificationComponent. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 24 HostGroups. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 ApiListener. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 CheckerComponent. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 4 Zones. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 2 Endpoints. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 ApiUser. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 2 Users. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 256 CheckCommands. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 2 UserGroups. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 7 ServiceGroups. [2018-12-08 12:55:13 +0000] information/ConfigItem: Instantiated 6 TimePeriods. [2018-12-08 12:55:13 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2018-12-08 12:55:13 +0000] information/cli: Finished validating the configuration file(s).