I have an event handler set up to restart a service when a particular check fails. It doesn't actually execute, no matter what I've tried. When I enable debuglog, I get log messages that it's executing the EventCommand, but the actual command never executes. I have used inotifywait to watch the directory with the script and nothing touches it, I have used strace to trace the Icinga2 daemon and it never tries to execute the command, it's as though the whole event handler system only exists as window-dressing with nothing underneath it. I even set up a test event handler to just log that it ran and nothing happens.
To Reproduce
Create event handler script:
# cat /usr/lib64/nagios/plugins/eventhandlers/test_event_handler
#!/usr/bin/bash
LOGFILE=/var/log/icinga2/test_event_handler.log
CMD="'$0'"
for a in "$@"
do
CMD="${CMD} '$a'"
done
echo "$$: `date`" >> ${LOGFILE}
echo "${CMD}" >> ${LOGFILE}
# ls -al /usr/lib64/nagios/plugins/eventhandlers/test_event_handler
-rwxr-xr-x. 1 root root 176 Jun 20 11:34 /usr/lib64/nagios/plugins/eventhandlers/test_event_handler
Create the event command:
object EventCommand "test_event" {
command = [ PluginDir + "/eventhandlers/test_event_hander" ]
arguments = {
"-s" = {
value = "$service.state$"
description = "State of the service being handled."
}
"-t" = {
value = "$service.state_type$"
description = "State type of the service being handled."
}
"-a" = {
value = "$service.check_attempt$"
description = "Check attempt of the service being handled."
}
}
}
Add the relevant pieces to the service definition:
event_command = "test_event"
enable_event_handler = true # should not be needed, as it defaults to true
Reload icinga2
Manually cause the service check to change state then look for the log file
Expected behavior
The event handler should run (in the above case, it should create/append to /var/log/icinga2/test_event_handler.log.
Your Environment
Running on RHEL 8 using the Icinga2 packages from the Icinga software repository subscription.
# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.2-1)
Copyright (c) 2012-2024 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Red Hat Enterprise Linux
Platform version: 8.10 (Ootpa)
Kernel: Linux
Kernel version: 4.18.0-513.24.1.el8_9.x86_64
Architecture: x86_64
Build information:
Compiler: GNU 8.5.0
Build host: unknown
OpenSSL version: OpenSSL 1.1.1k FIPS 25 Mar 2021
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf influxdb influxdb2 journald opentsdb perfdata syslog
Enabled features: api checker graphite icingadb livestatus mainlog notification
# icinga2 daemon -C
[2024-06-20 12:14:03 -0400] information/cli: Icinga application loader (version: r2.14.2-1)
[2024-06-20 12:14:03 -0400] information/cli: Loading configuration file(s).
[2024-06-20 12:14:03 -0400] information/ConfigItem: Committing config item(s).
[2024-06-20 12:14:03 -0400] information/ApiListener: My API identity: wamonitor2.iwco.com
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 NotificationComponent.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 LivestatusListener.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 2697 Dependencies.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 15 Users.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 13 UserGroups.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 3 TimePeriods.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 ServiceGroup.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 2634 Services.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 84 Zones.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 4 NotificationCommands.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 9446 Notifications.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 FileLogger.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 431 Hosts.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 27 HostGroups.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 2 EventCommands.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 82 Endpoints.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 Comment.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 3 ApiUsers.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 ApiListener.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 264 CheckCommands.
[2024-06-20 12:14:05 -0400] information/ConfigItem: Instantiated 1 IcingaDB.
[2024-06-20 12:14:05 -0400] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2024-06-20 12:14:05 -0400] information/cli: Finished validating the configuration file(s).
I do run a Master/Satellite/Server setup, but the affected service and event handler are all on the master.
Describe the bug
I have an event handler set up to restart a service when a particular check fails. It doesn't actually execute, no matter what I've tried. When I enable debuglog, I get log messages that it's executing the EventCommand, but the actual command never executes. I have used inotifywait to watch the directory with the script and nothing touches it, I have used strace to trace the Icinga2 daemon and it never tries to execute the command, it's as though the whole event handler system only exists as window-dressing with nothing underneath it. I even set up a test event handler to just log that it ran and nothing happens.
To Reproduce
Create the event command:
Expected behavior
The event handler should run (in the above case, it should create/append to /var/log/icinga2/test_event_handler.log.
Your Environment
Running on RHEL 8 using the Icinga2 packages from the Icinga software repository subscription.
I do run a Master/Satellite/Server setup, but the affected service and event handler are all on the master.