Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2.03k stars 578 forks source link

Icinga 2.9.0 won't run as Daemon on CentOS / RHEL 6 #6448

Closed nbuchwitz closed 6 years ago

nbuchwitz commented 6 years ago

I updated some old CentOS 6 servers and on one of them the Icinga 2 daemon only runs in foreground. If I use the the init script or even start the daemon via CLI with the -d flag no communication to the master is established nor any logs are written at run time:

[root@demeter.willi-graf.local][/var/log/icinga2]
# ls -l
total 8,0K
drwxr-x--- 3 icinga icingacmd 4,0K 17. Jul 15:03 compat/
drwxr-x--- 2 icinga icinga    4,0K 17. Jul 15:03 crash/
[root@demeter.willi-graf.local][/var/log/icinga2]
# bash -x /etc/init.d/icinga2 start
+ SYSCONFIGFILE=/etc/sysconfig/icinga2
+ '[' -f /etc/sysconfig/icinga2 ']'
+ . /etc/sysconfig/icinga2
+ : icinga
+ : icinga
+ : icingacmd
+ : /usr/sbin/icinga2
+ : /etc/icinga2/icinga2.conf
+ : /var/log/icinga2/error.log
+ : /var/log/icinga2/startup.log
+ : /var/run/icinga2/icinga2.pid
+ '[' -f /etc/default/icinga2 ']'
+ test -x /usr/sbin/icinga2
+ '[' '!' -e /etc/icinga2/icinga2.conf ']'
+ getent passwd icinga
+ getent group icinga
+ getent group icingacmd
+ '[' -f /etc/rc.d/init.d/functions ']'
+ . /etc/rc.d/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -f /etc/sysconfig/i18n -a -z '' -a -z '' ']'
++ . /etc/profile.d/lang.sh
++ unset LANGSH_SOURCED
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
+++ PROMPT=yes
+++ AUTOSWAP=no
+++ ACTIVE_CONSOLES='/dev/tty[1-6]'
+++ SINGLE=/sbin/sushell
++ '[' pty = serial ']'
++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
+++ cat /proc/cmdline
++ strstr 'ro root=/dev/mapper/vg_demeter-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_demeter/lv_swap rd_LVM_LV=vg_demeter/lv_root rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@48M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet console=ttyS0' rc.debug
++ '[' 'ro root=/dev/mapper/vg_demeter-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_demeter/lv_swap rd_LVM_LV=vg_demeter/lv_root rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@48M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet console=ttyS0' = 'ro root=/dev/mapper/vg_dem
eter-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_demeter/lv_swap rd_LVM_LV=vg_demeter/lv_root rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=129M@48M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet console=ttyS0' ']'
++ return 1
+ case "$1" in
+ checkconfig start fail
+ printf 'Checking configuration: '
Checking configuration: + /usr/sbin/icinga2 daemon -c /etc/icinga2/icinga2.conf -C
+ echo Done
Done
+ '[' xstart = x ']'
+ start
+ printf 'Starting Icinga 2: '
Starting Icinga 2: + /usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2
+ /usr/sbin/icinga2 daemon -c /etc/icinga2/icinga2.conf -d -e /var/log/icinga2/error.log
+ echo Done
Done
+ exit 0
[root@demeter.willi-graf.local][/var/log/icinga2]
# ls -l
total 8,0K
drwxr-x--- 3 icinga icingacmd 4,0K 17. Jul 15:03 compat/
drwxr-x--- 2 icinga icinga    4,0K 17. Jul 15:03 crash/
-rw-r--r-- 1 icinga icinga       0 18. Jul 10:43 debug.log
-rw------- 1 icinga icinga       0 18. Jul 10:43 error.log
-rw-r--r-- 1 icinga icinga       0 18. Jul 10:43 icinga2.log
-rw-r--r-- 1 root   root         0 18. Jul 10:43 startup.log

After stopping the daemon something is written in the logs files:

[root@demeter.willi-graf.local][/var/log/icinga2]
# service icinga2 stop
Stopping Icinga 2: .Done
[root@demeter.willi-graf.local][/var/log/icinga2]
# ls -l
total 16K
drwxr-x--- 3 icinga icingacmd 4,0K 17. Jul 15:03 compat/
drwxr-x--- 2 icinga icinga    4,0K 17. Jul 15:03 crash/
-rw-r--r-- 1 icinga icinga    2,3K 18. Jul 10:45 debug.log
-rw------- 1 icinga icinga       0 18. Jul 10:43 error.log
-rw-r--r-- 1 icinga icinga     607 18. Jul 10:45 icinga2.log
-rw-r--r-- 1 root   root         0 18. Jul 10:43 startup.log
root@demeter.willi-graf.local][/var/log/icinga2]
# cat icinga2.log
[2018-07-18 10:43:05 +0200] information/ApiListener: 'api' started.
[2018-07-18 10:43:05 +0200] information/ApiListener: Adding new listener on port '5665'
[2018-07-18 10:43:05 +0200] information/CheckerComponent: 'checker' started.
[2018-07-18 10:43:05 +0200] information/ConfigItem: Activated all objects.
[2018-07-18 10:45:56 +0200] information/Application: Received request to shut down.
[2018-07-18 10:45:56 +0200] information/Application: Shutting down...
[2018-07-18 10:45:56 +0200] information/ApiListener: 'api' stopped.
[2018-07-18 10:45:56 +0200] information/CheckerComponent: 'checker' stopped.
root@demeter.willi-graf.local][/var/log/icinga2]
# cat debug.log
[2018-07-18 10:43:05 +0200] information/ApiListener: 'api' started.
[2018-07-18 10:43:05 +0200] information/ApiListener: Adding new listener on port '5665'
[2018-07-18 10:43:05 +0200] information/CheckerComponent: 'checker' started.
[2018-07-18 10:43:05 +0200] information/ConfigItem: Activated all objects.
[2018-07-18 10:43:05 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonCommand::Run'
[2018-07-18 10:43:05 +0200] debug/IcingaApplication: In IcingaApplication::Main()
[2018-07-18 10:43:05 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:43:20 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:43:35 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:43:50 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:44:05 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:44:20 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:44:35 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:44:50 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:45:05 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:45:20 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:45:35 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:45:50 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0%
[2018-07-18 10:45:56 +0200] information/Application: Received request to shut down.
[2018-07-18 10:45:56 +0200] information/Application: Shutting down...
[2018-07-18 10:45:56 +0200] information/ApiListener: 'api' stopped.
[2018-07-18 10:45:56 +0200] information/CheckerComponent: 'checker' stopped.

My /etc/sysconfig/icinga2 looks like the following:

#This is the default environment Icinga 2 runs with.
#Make your changes here.

#DAEMON=/usr/sbin/icinga2
#ICINGA2_CONFIG_FILE=/etc/icinga2/icinga2.conf
#ICINGA2_RUN_DIR=/var/run
#ICINGA2_STATE_DIR=/var
#ICINGA2_PID_FILE=/var/run/icinga2/icinga2.pid
#ICINGA2_LOG_DIR=/var/log/icinga2
#ICINGA2_ERROR_LOG=/var/log/icinga2/error.log
#ICINGA2_STARTUP_LOG=/var/log/icinga2/startup.log
#ICINGA2_LOG=/var/log/icinga2/icinga2.log
#ICINGA2_CACHE_DIR=/var/cache/icinga2
#ICINGA2_USER=icinga
#ICINGA2_GROUP=icinga
#ICINGA2_COMMAND_GROUP=icingacmd

A downgrade to Icinga 2.8.4 works without any problems on the same machines. Do I miss something?

Your Environment

Copyright (c) 2012-2018 Icinga Development Team (https://www.icinga.com/) License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Application information: Installation root: /usr Sysconf directory: /etc Run directory: /var/run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /var/run/icinga2/icinga2.pid

System information: Platform: CentOS Platform version: 6.10 (Final) Kernel: Linux Kernel version: 2.6.32-754.2.1.el6.x86_64 Architecture: x86_64

Build information: Compiler: GNU 4.8.2 Build host: 5009d75245b1

* Enabled features (`icinga2 feature list`):

Disabled features: command compatlog elasticsearch gelf graphite influxdb livestatus notification opentsdb perfdata statusdata syslog Enabled features: api checker debuglog mainlog

* Config validation (`icinga2 daemon -C`):

[2018-07-18 10:51:51 +0200] information/cli: Icinga application loader (version: r2.9.0-1) [2018-07-18 10:51:51 +0200] information/cli: Loading configuration file(s). [2018-07-18 10:51:51 +0200] information/ConfigItem: Committing config item(s). [2018-07-18 10:51:51 +0200] information/ApiListener: My API identity: demeter.willi-graf.local [2018-07-18 10:51:51 +0200] information/ConfigItem: Instantiated 2 FileLoggers. [2018-07-18 10:51:51 +0200] information/ConfigItem: Instantiated 1 IcingaApplication. [2018-07-18 10:51:51 +0200] information/ConfigItem: Instantiated 1 ApiListener. [2018-07-18 10:51:51 +0200] information/ConfigItem: Instantiated 1 CheckerComponent. [2018-07-18 10:51:51 +0200] information/ConfigItem: Instantiated 2 Zones. [2018-07-18 10:51:51 +0200] information/ConfigItem: Instantiated 2 Endpoints. [2018-07-18 10:51:51 +0200] information/ConfigItem: Instantiated 213 CheckCommands. [2018-07-18 10:51:51 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2018-07-18 10:51:51 +0200] information/cli: Finished validating the configuration file(s).

Mikesch-mp commented 6 years ago

~~Did you remove /etc/sysconfig/icinga2 before upgrade? If there is any variable $variable it will not work, see sysconfig changes in the upgrade docs~~

Just found your sysconfig file is clean :)

dnsmichi commented 6 years ago

This system works as a client, right? Can you share the zones.conf and any modification different from a fresh installation?

nbuchwitz commented 6 years ago

Besides the zones.conf nothing is customised.

As requested in the other issue I saved the gdb output: https://gist.github.com/nbuchwitz/733765116b90c29a9eb9ced8f90a54ea

zones.conf

/*
 * Generated by Icinga 2 node setup commands
 * on 2017-08-15 15:49:01 +0200
 */

object Endpoint "monitoring.remote.fqdn" {
        host = "10.0.0.4"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "monitoring.remote.fqdn" ]
}

object Endpoint NodeName {
}

object Zone ZoneName {
        endpoints = [ NodeName ]
        parent = "master"
}

icinga2.conf

include "constants.conf"
include "zones.conf"

include <itl>
include <plugins>
include <plugins-contrib>
include <manubulon>
include <windows-plugins>
include <nscp>

include "features-enabled/*.conf"

constants.conf

 * This file defines global constants which can be used in
 * the other configuration files.
 */

/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = "/usr/lib64/nagios/plugins"

/* The directory which contains the Manubulon plugins.
 * Check the documentation, chapter "SNMP Manubulon Plugin Check Commands", for details.
 */
const ManubulonPluginDir = "/usr/lib64/nagios/plugins"

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.
 * Check the documentation, chapter "Plugins Contribution", for details.
 */
const PluginContribDir = "/usr/lib64/nagios/plugins"

/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "demeter.willi-graf.local"

/* Our local zone name. */
const ZoneName = "demeter.willi-graf.local"

/* Secret key for remote node tickets */
const TicketSalt = ""
Al2Klimov commented 6 years ago

Confirmed. Set up v2.9 on CentOS 6 as usual with the node wizard and... nothing happens.

gskjold commented 6 years ago

I can also confirm. I have a mixed CentOS 6 and 7 environment. All CentOS 7 server clients are connected, while all CentOS 6 servers have lost connection. They were upgraded monday night 0400 CEST, and I could swear they all worked yesterday.

Forgot I have auto upgrade on these. Looking in logs, they we're upgraded 18.07.18 04.00 CEST. Looks like they broke instantly on upgrade to 2.9.0-1

dnsmichi commented 6 years ago

I'm closing this in favour of #6445 - please continue there.

dnsmichi commented 6 years ago

Final reference: https://github.com/Icinga/icinga2/issues/6445#issuecomment-406250387