Unable to send notification from icinga2 cluster instance

[rajeshjanjam]

Hi All,

Problem is, that when host or service is down, Icinga not sending any emails. Postfix is running and i can send email to me using mail command through command line is just fine.

My Environment

• Version used: icinga2 - The Icinga 2 network monitoring daemon (version: v2.6.0)

• Operating System and version: Cent OS7

• Cluster Set-Up Type : (Top Down Config Sync approach)

• Enabled features (icinga2 feature list):

  Master Instance : Disabled features: gelf graphite influxdb livestatus opentsdb syslog Enabled features: api checker command compatlog debuglog ido-mysql mainlog notification perfdata statusdata

Client Instance : Disabled features: gelf graphite influxdb livestatus opentsdb syslog Enabled features: api checker command compatlog debuglog ido-mysql mainlog notification perfdata statusdata

• Icinga Web 2 version and modules (System - About): Version 2.4.1 and module is "monitoring"

• Config validation (icinga2 daemon -C):

information/cli: Icinga application loader (version: v2.6.0) information/cli: Loading configuration file(s). information/ConfigItem: Committing config item(s). information/ApiListener: My API identity: wargaming-icinga-node2 information/ConfigItem: Instantiated 1 ApiListener. information/ConfigItem: Instantiated 2 Zones. information/ConfigItem: Instantiated 2 FileLoggers. information/ConfigItem: Instantiated 2 Endpoints. information/ConfigItem: Instantiated 1608 Notifications. information/ConfigItem: Instantiated 5 NotificationCommands. information/ConfigItem: Instantiated 208 CheckCommands. information/ConfigItem: Instantiated 1 IcingaApplication. information/ConfigItem: Instantiated 1 EventCommand. information/ConfigItem: Instantiated 15 Hosts. information/ConfigItem: Instantiated 15 Users. information/ConfigItem: Instantiated 4 UserGroups. information/ConfigItem: Instantiated 776 Dependencies. information/ConfigItem: Instantiated 4 TimePeriods. information/ConfigItem: Instantiated 1 ServiceGroup. information/ConfigItem: Instantiated 812 Services. information/ConfigItem: Instantiated 1 CompatLogger. information/ConfigItem: Instantiated 1 StatusDataWriter. information/ConfigItem: Instantiated 1 ExternalCommandListener. information/ConfigItem: Instantiated 1 CheckerComponent. information/ConfigItem: Instantiated 1 IdoMysqlConnection. information/ConfigItem: Instantiated 1 NotificationComponent. information/ConfigItem: Instantiated 1 PerfdataWriter. information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' information/cli: Finished validating the configuration file(s).

• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

  object Endpoint server1{ host = "10.10.xx.xx" log_duration = 0m }

object Endpoint "server2" { host = "10.10.xx.xx" log_duration = 0m }

object Zone ZoneName { endpoints = [ sever1, "server2" ] }

object Zone "global-templates" { global = true }

• Command : pcs status Cluster name: webcluster Stack: corosync Current DC: Server1 (version 1.1.15-11.el7_3.2-e174ec8) - partition with quorum Last updated: Thu Jun 29 12:10:02 2017 Last change: Thu Jun 29 12:08:25 2017 by root via cibadmin on Server2

  2 nodes and 2 resources configured

  Node Server1: standby Online: [ Server2 ]

  Full list of resources:

  Cluster_VIP (ocf::heartbeat:IPaddr2): Started Server2 WebServer (ocf::heartbeat:apache): Started Server2

  Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled

Configuration Definitions :

object Host "Warships" { import "generic-host" address = "warships.com" }

object Service "Warships" { import "generic-service" host_name = "Warships" check_command = "check_http" display_name = "Warships" vars.ARG1 = "warships.com" vars.ARG2 = "http://warships.com/" vars.ARG3 = "60" vars.URL_type = "Websites" }

object CheckCommand "check_http" { import "plugin-check-command" command = [ PluginDir + "/check_http" ] arguments = { "-I" = "$address$" "-H" = "$ARG1$" "-u" = "$ARG2$" "-t" = "$ARG3$" "-p" = "$ARG4$" "-R" = "$ARG5$" "-w" = "$ARG6$" "-c" = "$ARG7$" "-a" = "$ARG8$" "-C" = "$ARG9$" "--string" = "$ARG10$" "-k" = "$ARG11$" "-r" = "$ARG12$" "--header" = "$ARG13$" "-e" = "$ARG14$" "-A" = "$ARG15$" "-R" = "$ARG16$" "-S" = { set_if = "$http_ssl$" } } vars.http_ssl = false

}

template Host "generic-host" { max_check_attempts = 3 check_interval = 5m retry_interval = 30s check_command = "hostalive" }

template Service "generic-service" { max_check_attempts = 5 check_interval = 5m retry_interval = 30s }

object NotificationCommand "migration-notification-command" { import "plugin-notification-command" vars.USER1 = PluginDir }

object User "Shan" { import "generic-contact" email = "xxxx@gmail.com" }

object User "Rajesh" { import "generic-contact" email = "xxxx@gmail.com" }

object User "Raju" { import "generic-contact" email = "xxxx@gmail.com" }

object UserGroup "Test" {

assign where user.name == "Shan"
assign where user.name == "Rajesh"
assign where user.name == "Raju"

}

apply Notification "mail-icingaadmin" to Host { import "mail-host-notification"

user_groups = [ "Test" ] assign where host.name }

apply Notification "mail-icingaadmin" to Service { import "mail-service-notification"

user_groups = [ "global-admins", "Test" ] assign where service.name }

template Notification "mail-host-notification" { command = "mail-host-notification"

states = [ Up, Down ] types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]

period = "24x7" }

template Notification "mail-service-notification" { command = "mail-service-notification"

states = [ OK, Warning, Critical, Unknown ] types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]

period = "24x7" }

object NotificationCommand "Service-All Links-notification-command-notify-service-by-email" { import "migration-notification-command" import "mail-service-notification" }

apply Notification "notification-Service-All Links-notification-command-notify-service-by-email" to Service { import "generic-service-notification" command = "Service-All Links-notification-command-notify-service-by-email" period = "24x7" interval = 60m states = [ OK, Critical, Unknown, Warning ] types = [ Problem, Custom, Recovery ] user_groups = [ "Test" ]

assign where service.vars.individual_url_type == "Websites"

}

Icinga2 Log :

[2017-06-29 13:53:28 -0400] information/Checkable: Checking for configured notifications for object 'warships' [2017-06-29 13:53:28 -0400] information/IdoMysqlConnection: Query queue items: 0, query rate: 14.25/s (855/min 1786/5min 2252/15min); [2017-06-29 13:53:43 -0400] information/IdoMysqlConnection: Query queue items: 1, query rate: 14.3667/s (862/min 1795/5min 2320/15min); [2017-06-29 13:53:52 -0400] information/ExternalCommandListener: Executing external command: [1498758832] SEND_CUSTOM_HOST_NOTIFICATION;Warships;2;icingaadmin;Testing

Debug Log:

[2017-06-29 13:53:52 -0400] debug/IdoMysqlConnection: Query: COMMIT [2017-06-29 13:53:52 -0400] debug/IdoMysqlConnection: Query: BEGIN [2017-06-29 13:53:52 -0400] information/ExternalCommandListener: Executing external command: [1498758832] SEND_CUSTOM_HOST_NOTIFICATION;Warships;2;icingaadmin;Testing [2017-06-29 13:53:52 -0400] debug/DbEvents: add external command history [2017-06-29 13:53:52 -0400] notice/ExternalCommandProcessor: Sending custom notification for host Warships [2017-06-29 13:53:52 -0400] notice/ApiListener: Relaying 'event::SetForceNextNotification' message [2017-06-29 13:53:52 -0400] information/Checkable: Checking for configured notifications for object 'Warships' [2017-06-29 13:53:52 -0400] debug/Checkable: Checkable 'Warships' has 1 notification(s). [2017-06-29 13:53:52 -0400] notice/ApiListener: Sending message 'event::SetForceNextNotification' to 'server2' [2017-06-29 13:53:52 -0400] notice/ApiListener: Relaying 'event::SendNotifications' message [2017-06-29 13:53:52 -0400] notice/ApiListener: Sending message 'event::SendNotifications' to 'server2' [2017-06-29 13:53:52 -0400] notice/ApiListener: Relaying 'event::SetForceNextNotification' message [2017-06-29 13:53:52 -0400] notice/ApiListener: Sending message 'event::SetForceNextNotification' to 'server2' [2017-06-29 13:53:52 -0400] notice/JsonRpcConnection: Received 'event::SetForceNextNotification' message from 'server2' [2017-06-29 13:53:52 -0400] notice/ApiListener: Relaying 'event::SetForceNextNotification' message [2017-06-29 13:53:52 -0400] notice/JsonRpcConnection: Received 'event::NotificationSentToAllUsers' message from 'server2' [2017-06-29 13:53:52 -0400] notice/ApiListener: Relaying 'event::NotificationSentToAllUsers' message [2017-06-29 13:53:52 -0400] debug/DbEvents: add notification history for 'Warships' [2017-06-29 13:53:52 -0400] debug/DbEvents: add contact notification history for service 'Warships' and user 'Raju'. [2017-06-29 13:53:52 -0400] debug/DbEvents: add contact notification history for service 'Warships' and user 'Rajesh'. [2017-06-29 13:53:52 -0400] debug/DbEvents: add contact notification history for service 'Warships' and user 'Shan'. Mail Log :

Note: When I force check to send custom notification mails are getting triggered and receiving to user end but without force check custom notification mails are not getting trigger.

Jun 29 13:53:58 server2 postfix/smtp[3301]: 8049DC015B12: to=xxxxx@gmail.com, relay=10.10.10.90[10.10.10.90]:25, delay=0.37, delays=0.31/0.03/0/0.03, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 98BA62499FF) Jun 29 13:53:58 server2 postfix/qmgr[3152]: 8049DC015B12: removed Jun 29 13:53:58 server2 postfix/smtp[3302]: 8EF53C015B13: to=xxxxxx@gmail.com, relay=10.10.10.90[10.10.10.90]:25, delay=0.37, delays=0.32/0.03/0/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 99D1B249A3D) Jun 29 13:53:58 server2 postfix/qmgr[3152]: 8EF53C015B13: removed Jun 29 13:53:58 server2 postfix/smtp[3303]: 91A1CC015B53: to=xxxxx@gmail.com, relay=10.10.10.90[10.10.10.90]:25, delay=0.37, delays=0.32/0.03/0/0.02, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 9ACB5249A4C) Jun 29 13:53:58 server2 postfix/qmgr[3152]: 91A1CC015B53: removed

Expected Behavior

When host or service gets down email should get triggered and when we trigger custom notification for any host or service it should notification without forcing the notification.

Current Behavior

Custom notifications are alone being triggered when we force check and in rest of the times when a host or service gets down no notifications are being sent.

Possible Solution tried

• We tried custom notifications from icingaweb2, by start and stopping "postfix", "send mail" services through command line and tried to figure out mail log.
• Checked for error in icinga2 log as well but its clean.
• Verified configuration errors did check config before reloading icinga2.
• Stopped cluster and tried to make one of the instance as standalone and tried sending custom notification as well but until forced the notification to send , no notifications were received to user.

Please check the above configuration set up and let me known for any further info needed. Any help on this would be appreciated. Thanks!!

[Stefar77]

try adding states and types to users and I think Notification. Direct of via templates.

apply Notification "mail-icingaadmin" to Host {
  states = [ Down, Up ]
  types = [ Acknowledgement, Custom, Problem, Recovery ]

and

object User "Raju" {
    states = [ Critical, Down, OK, Up, Warning ]
    types = [ Acknowledgement, Custom, Problem, Recovery ]

[Crunsher]

Hello, are you still experiencing this issue? Have you tried @Stefar77s solution?

[dnsmichi]

No feedback received.