Does not execute notification commands?

[kam1sh]

​I'm having an issue with setting up notifications in Icinga 2. Notification successfuly applies to services, and users have email fields and they could receive notifications, i have custom notification command, but the problem is not in it, notification command just doesn't even execute. All code and info are below. Also, Icinga Web 2 shows my contact some.user under button "Send notification" in the service description. It just doesn't send anything

The problem is: when i execute custom notification in Icinga Web 2 or the service in warn/crit state, icinga does not execute NotificationCommand (and don't even write this in icinga log (and debug log too)). It write nothing about this notification at all. Also, i'm using Icinga Director, but i wrote it here, because the problem in icinga itself.

​​object User "some.user" {
​    enable_notifications = true
    display_name = "some.user"
    email = "<there was some mail>"
    states = [ Critical, OK, Unknown, Warning ]
    types = [ Acknowledgement, Custom, Problem, Recovery ]
}
​template Host "NRPE Agent" {
    check_command = "hostalive4"
    vars.nrpe = true
}
​object Host "testhost" {
    import "NRPE Agent"
    display_name = "TEST"
    address = "<there was some ip addr>"
    vars.nrpe = true
}
​object NotificationCommand "mail-service-notification-new" {
    import "plugin-notification-command"
    command = [ "/etc/icinga2/scripts/notification-to-email.sh" ]
    arguments += {
        "--host-address" = "$address$"
        "--host-alias" = "$host.name$"
        "--host-displayname" = "$host.display_name$"
        "--host-output" = "$host.output$"
        "--host-state" = "$host.state$"
        "--icingaweb-url" = "<there was some url>"
        "--long-datetime" = "$icinga.long_date_time$"
        "--mail-from" = "<there was another email>"
        "--mail-to" = "$user.email$"
        "--notification-comment" = "$notification.comment$"
        "--notification-type" = "$notification.type$"
        "--plugin-mode" = "service-mode"
        "--service-desc" = "$service.name$"
        "--service-displayname" = "$service.display_name$"
        "--service-output" = "$service.output$"
        "--service-state" = "$service.state$"
    }
}
​template Notification "E-mail notifications for services 24x5" {
    command = "mail-service-notification-new"
    states = [ Critical, OK, Unknown, Warning ]
    types = [ Acknowledgement, Custom, Problem, Recovery ]
}
​​object TimePeriod "Work days" {
    import "legacy-timeperiod"
    display_name = "Always"
    ranges = {
        "friday"    = "00:00-24:00"
        "monday"    = "00:00-24:00"
        "thursday"    = "00:00-24:00"
        "tuesday"    = "00:00-24:00"
        "wednesday"    = "00:00-24:00"
    }
}
​apply Notification "Test notification" to Service {
    import "E-mail notifications for services 24x7"
    period = "Work days"
    assign where host.display_name == "TEST"
    users = [ "some.user" ]
}

[root@icinga2 ~]# icinga2 daemon -C information/cli: Icinga application loader (version: v2.6.2) information/cli: Loading configuration file(s). information/ConfigItem: Committing config item(s). information/ApiListener: My API identity: icinga2 warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere! information/ConfigItem: Instantiated 1 ApiUser. information/ConfigItem: Instantiated 1 ApiListener. information/ConfigItem: Instantiated 3 Zones. information/ConfigItem: Instantiated 1 FileLogger. information/ConfigItem: Instantiated 2 Endpoints. information/ConfigItem: Instantiated 30 Notifications. information/ConfigItem: Instantiated 3 NotificationCommands. information/ConfigItem: Instantiated 1 ScheduledDowntime. information/ConfigItem: Instantiated 201 CheckCommands. information/ConfigItem: Instantiated 1 Downtime. information/ConfigItem: Instantiated 2 HostGroups. information/ConfigItem: Instantiated 1 IcingaApplication. information/ConfigItem: Instantiated 3 Hosts. information/ConfigItem: Instantiated 3 Users. information/ConfigItem: Instantiated 1 UserGroup. information/ConfigItem: Instantiated 4 TimePeriods. information/ConfigItem: Instantiated 3 ServiceGroups. information/ConfigItem: Instantiated 39 Services. information/ConfigItem: Instantiated 1 CheckerComponent. information/ConfigItem: Instantiated 1 GraphiteWriter. information/ConfigItem: Instantiated 1 IdoMysqlConnection. information/ConfigItem: Instantiated 1 NotificationComponent. information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' information/cli: Finished validating the configuration file(s).​

[root@icinga2 ~]# icinga2 feature list Disabled features: command compatlog debuglog gelf influxdb livestatus opentsdb perfdata statusdata syslog Enabled features: api checker graphite ido-mysql mainlog notification

[root@icinga2 ~]# icinga2 --version icinga2 - The Icinga 2 network monitoring daemon (version: v2.6.2) ​<...> System information: Platform: CentOS Linux Platform version: 7 (Core) Kernel: Linux Kernel version: 3.10.0-514.el7.x86_64 Architecture: x86_64

​Icinga Director version is 1.3.1​ Icinga Web 2 version is 2.4.1​

[dnsmichi]

Enable the debug log and check that the given host changes to a HARD state thus triggering the notification event. Post your findings here.

[kam1sh]

Yes! With hard state it works! [2017-03-24 16:00:38 +0300] notice/Process: Running command '/etc/icinga2/scripts/notification-to-email.sh' '--host-address' '<...> '--host-alias' '<...>' '--host-displayname' 'TEST' '--host-output' 'PING OK - Packet loss = 0%, RTA = 45.94 ms' '--host-state' 'UP' '--icingaweb-url' '<...>' '--long-datetime' '2017-03-24 16:00:38 +0300' '--mail-from' '<...> '--mail-to' '<...>' '--notification-comment' '' '--notification-type' 'PROBLEM' '--plugin-mode' 'service-mode' '--service-desc' 'DISK' '--service-displayname' 'DISK' '--service-output' 'DISK CRITICAL - free space: / 2975 MB (66% inode=97%); ' '--service-state' 'CRITICAL': PID 24178 [2017-03-24 16:00:38 +0300] debug/DbEvents: add log entry history for '<...>!DISK' But i still don't understand why it doesn't send custom notifications. I dunno, maybe it's a problem of icinga web?

[dnsmichi]

Ok, and if you send in a custom notification command, trace the log lines down and post them here.

[kam1sh]

icinga web 2:

2017-03-28T13:46:54+03:00 - DEBUG - Sending external Icinga command "[1490698014] SEND_CUSTOM_SVC_NOTIFICATION;;CPU;2;<...>;another test" to the local command file "/var/run/icinga2/cmd/icinga2.cmd"

icinga 2:

[2017-03-28 13:46:54 +0300] information/Notification: Sending 'Custom' notification '!CPU!<...> notification for user '<...>' [2017-03-28 13:46:54 +0300] information/Notification: Completed sending 'Custom' notification '!CPU!<...> notification' for checkable '!CPU' and user '<...>'.

OK, i think the problem was in some disabled features. Right now i have enabled notifications and commands, and everything is good.