search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2.03k stars 578 forks source link

reload icinga2 via systemctl doesnt work anymore, restarts works. #6434

Closed Mikesch-mp closed 6 years ago

Mikesch-mp commented 6 years ago

With the latest snapshot build, icinga2 cant reload anymore, only restart works.

Expected Behavior

reload of icinga shoould work like it works before when there is no configuration error

Current Behavior

reload is not possible, systemctl reports problems

Possible Solution

Steps to Reproduce (for bugs)

Install latest snaphot build on Centos (7.5.1804)

Context

Whilte testing the latest packages, i did also some config changes. When i tried to activate them via systemctl reload icinga2 it always fails.

Your Environment

Copyright (c) 2012-2018 Icinga Development Team (https://www.icinga.com/) License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Application information: Installation root: /usr Sysconf directory: /etc Run directory: /run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid

System information: Platform: CentOS Linux Platform version: 7 (Core) Kernel: Linux Kernel version: 3.10.0-862.6.3.el7.x86_64 Architecture: x86_64

Build information: Compiler: GNU 4.8.5 Build host: unknown


* Operating System and version:

[root@koelin01 hosts]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core)


* Enabled features (`icinga2 feature list`):

[root@koelin01 hosts]# icinga2 feature list Disabled features: command compatlog gelf livestatus opentsdb perfdata statusdata syslog Enabled features: api checker debuglog elasticsearch graphite ido-mysql influxdb mainlog notification


* Config validation (`icinga2 daemon -C`):

[root@koelin01 _]# icinga2 daemon -C [2018-07-16 09:55:52 +0200] information/cli: Icinga application loader (version: v2.8.4-816-g8c8f1de) [2018-07-16 09:55:52 +0200] information/cli: Loading configuration file(s). [2018-07-16 09:55:52 +0200] information/ConfigItem: Committing config item(s). [2018-07-16 09:55:52 +0200] information/ApiListener: My API identity: koelin01.fritz.box [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 123 Services. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 3 ServiceGroups. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 InfluxdbWriter. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 9 HostGroups. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 2 FileLoggers. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 NotificationComponent. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 2 NotificationCommands. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 187 Notifications. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 IcingaApplication. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 64 Hosts. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 ApiListener. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 Downtime. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 ElasticsearchWriter. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 GraphiteWriter. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 CheckerComponent. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 6 Zones. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 4 Endpoints. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 6 ApiUsers. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 UserGroup. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 221 CheckCommands. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 5 TimePeriods. [2018-07-16 09:55:53 +0200] information/ConfigItem: Instantiated 2 Users. [2018-07-16 09:55:53 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2018-07-16 09:55:53 +0200] information/cli: Finished validating the configuration file(s).


Syslog messages for reload:

Jul 16 09:48:02 koelin01 safe-reload: Validating config files: Done Jul 16 09:48:02 koelin01 systemd: icinga2.service: control process exited, code=exited status=7 Jul 16 09:48:02 koelin01 safe-reload: Reloading Icinga 2: Jul 16 09:48:02 koelin01 systemd: Reload failed for Icinga host/service/network monitoring system.

Output of restart via systemctl with loglevel set to debug:

[root@koelin01 _]# export SYSTEMD_LOGLEVEL=debug [root@koelin01 ]# systemctl reload icinga2 Failed to determine peer security context: Protocol not available Calling manager for ReloadUnit on icinga2.service, replace Got unexpected auxiliary data with level=1 and type=2 Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=ReloadUnit cookie=1 reply_cookie=0 error=n/a Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnit cookie=2 reply_cookie=0 error=n/a Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1/unit/icinga2_2eservice interface=org.freedesktop.DBus.Properties member=Get cookie=3 reply_cookie=0 error=n/a Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Adding /org/freedesktop/systemd1/job/10084 to the set Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobNew cookie=2 reply_cookie=0 error=n/a Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/icinga2_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=3 reply_cookie=0 error=n/a Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/unit/icinga2_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=4 reply_cookie=0 error=n/a Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1/job/10084 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=5 reply_cookie=0 error=n/a Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=8 reply_cookie=0 error=n/a Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1/unit/icinga2_2eservice interface=org.freedesktop.DBus.Properties member=Get cookie=4 reply_cookie=0 error=n/a Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Got unexpected auxiliary data with level=1 and type=2 Job for icinga2.service failed because the control process exited with error code. See "systemctl status icinga2.service" and "journalctl -xe" for details. Got result failed/Input/output error for job icinga2.service

dnsmichi commented 6 years ago

I'm not able to reproduce this.

michi@mbmif ~/coding/icinga/icinga-vagrant/standalone (master=) $ vagrant ssh
sudo -i
Last login: Mon Jul 16 10:08:34 2018
sudo -i
 _____     _
|_   _|   (_)
  | |  ___ _ _ __   __ _  __ _
  | | / __| | '_ \ / _` |/ _` |
 _| || (__| | | | | (_| | (_| |
|_____\___|_|_| |_|\__, |\__,_|
                    __/ |
                   |___/
[vagrant@icinga2 ~]$ sudo -i
[root@icinga2 ~]# systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-07-16 10:17:45 CEST; 10min ago
  Process: 28174 ExecStartPre=/usr/lib/icinga2/prepare-dirs (code=exited, status=0/SUCCESS)
 Main PID: 28183 (icinga2)
   CGroup: /system.slice/icinga2.service
           ├─ 1700 /bin/perl -w /usr/lib64/nagios/plugins/check_mysql_health --database icinga --hostname 1.2.3.4 --mode connection-time --password...
           ├─ 2215 /usr/lib64/nagios/plugins/check_ping -4 -H 127.0.0.1 -c 200,15% -w 100,5%
           ├─ 2216 /usr/bin/ping -n -U -W 10 -c 5 127.0.0.1
           ├─ 2217 /usr/lib64/nagios/plugins/check_ping -4 -H 1.2.3.4 -c 200,15% -w 100,5%
           ├─ 2218 /usr/bin/ping -n -U -W 10 -c 5 1.2.3.4
           ├─ 2219 /usr/lib64/nagios/plugins/check_ping -4 -H 1.2.3.9 -c 200,15% -w 100,5%
           ├─ 2220 /usr/bin/ping -n -U -W 10 -c 5 1.2.3.9
           ├─ 2239 /usr/lib64/nagios/plugins/check_http -I 1.2.3.9 -u /icingaweb2
           ├─28183 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e
           └─28218 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e

Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:40 +0200] information/Checkable: Checking for configured no...map-1'
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:40 +0200] information/Checkable: Checkable 'map-server-vien...tions.
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:40 +0200] information/Checkable: Checking for configured no...m-004'
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:40 +0200] information/Notification: Sending 'Problem' notif...admin'
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:40 +0200] information/Notification: Completed sending 'Prob...dmin'.
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:41 +0200] information/ApiListener: New client connection fr...icate)
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:41 +0200] information/HttpServerConnection: Request: GET /v...shing)
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:41 +0200] information/ApiListener: New client connection fr...icate)
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:41 +0200] information/HttpServerConnection: Request: GET /v...shing)
Jul 16 10:27:41 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:41 +0200] information/ApiListener: New client connection fr...icate)
Hint: Some lines were ellipsized, use -l to show in full.
[root@icinga2 ~]# systemctl reload icinga2
[root@icinga2 ~]# systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-07-16 10:17:45 CEST; 10min ago
  Process: 2301 ExecReload=/usr/lib/icinga2/safe-reload (code=exited, status=0/SUCCESS)
  Process: 28174 ExecStartPre=/usr/lib/icinga2/prepare-dirs (code=exited, status=0/SUCCESS)
 Main PID: 2346 (icinga2)
   CGroup: /system.slice/icinga2.service
           ├─2217 /usr/lib64/nagios/plugins/check_ping -4 -H 1.2.3.4 -c 200,15% -w 100,5%
           ├─2218 /usr/bin/ping -n -U -W 10 -c 5 1.2.3.4
           ├─2219 /usr/lib64/nagios/plugins/check_ping -4 -H 1.2.3.9 -c 200,15% -w 100,5%
           ├─2220 /usr/bin/ping -n -U -W 10 -c 5 1.2.3.9
           ├─2239 /usr/lib64/nagios/plugins/check_http -I 1.2.3.9 -u /icingaweb2
           ├─2346 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e --reload-internal 28183
           └─2363 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e --reload-internal 28183

Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:54 +0200] information/Notification: Sending 'Problem' notif...admin'
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:54 +0200] information/Notification: Completed sending 'Prob...dmin'.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #8 (IdoMysqlConnection, id...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #6 (ApiListener, SyncQueue...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #7 (GraphiteWriter, graphi...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #5 (ApiListener, RelayQueu...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Application: Got reload command: Star...tance.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Checkable: Checking for configured no...map-0'
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Checkable: Checkable 'map-server-rio-...tions.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Application: Reload requested, lettin... over.
Hint: Some lines were ellipsized, use -l to show in full.
[root@icinga2 ~]# systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-07-16 10:17:45 CEST; 10min ago
  Process: 2301 ExecReload=/usr/lib/icinga2/safe-reload (code=exited, status=0/SUCCESS)
  Process: 28174 ExecStartPre=/usr/lib/icinga2/prepare-dirs (code=exited, status=0/SUCCESS)
 Main PID: 2346 (icinga2)
   CGroup: /system.slice/icinga2.service
           ├─2346 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e --reload-internal 28183
           └─2363 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e --reload-internal 28183

Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:54 +0200] information/Notification: Sending 'Problem' notif...admin'
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:54 +0200] information/Notification: Completed sending 'Prob...dmin'.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #8 (IdoMysqlConnection, id...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #6 (ApiListener, SyncQueue...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #7 (GraphiteWriter, graphi...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #5 (ApiListener, RelayQueu...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Application: Got reload command: Star...tance.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Checkable: Checking for configured no...map-0'
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Checkable: Checkable 'map-server-rio-...tions.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Application: Reload requested, lettin... over.
Hint: Some lines were ellipsized, use -l to show in full.
[root@icinga2 ~]# systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-07-16 10:17:45 CEST; 10min ago
  Process: 2301 ExecReload=/usr/lib/icinga2/safe-reload (code=exited, status=0/SUCCESS)
  Process: 28174 ExecStartPre=/usr/lib/icinga2/prepare-dirs (code=exited, status=0/SUCCESS)
 Main PID: 2346 (icinga2)
   CGroup: /system.slice/icinga2.service
           ├─2346 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e --reload-internal 28183
           └─2363 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e --reload-internal 28183

Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:54 +0200] information/Notification: Sending 'Problem' notif...admin'
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:54 +0200] information/Notification: Completed sending 'Prob...dmin'.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #8 (IdoMysqlConnection, id...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #6 (ApiListener, SyncQueue...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #7 (GraphiteWriter, graphi...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #5 (ApiListener, RelayQueu...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Application: Got reload command: Star...tance.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Checkable: Checking for configured no...map-0'
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Checkable: Checkable 'map-server-rio-...tions.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Application: Reload requested, lettin... over.
Hint: Some lines were ellipsized, use -l to show in full.
[root@icinga2 ~]# systemctl reload icinga2
[root@icinga2 ~]# systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-07-16 10:17:45 CEST; 10min ago
  Process: 2450 ExecReload=/usr/lib/icinga2/safe-reload (code=exited, status=0/SUCCESS)
  Process: 28174 ExecStartPre=/usr/lib/icinga2/prepare-dirs (code=exited, status=0/SUCCESS)
 Main PID: 2495 (icinga2)
   CGroup: /system.slice/icinga2.service
           ├─2429 /usr/lib64/nagios/plugins/check_ping -4 -H 1.2.3.4 -c 200,15% -w 100,5%
           ├─2430 /usr/bin/ping -n -U -W 10 -c 5 1.2.3.4
           ├─2495 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e --reload-internal 2346
           └─2512 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e --reload-internal 2346

Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #7 (GraphiteWriter, graphi...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:55 +0200] information/WorkQueue: #5 (ApiListener, RelayQueu...5min);
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Application: Got reload command: Star...tance.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Checkable: Checking for configured no...map-0'
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Checkable: Checkable 'map-server-rio-...tions.
Jul 16 10:27:56 icinga2.vagrant.demo.icinga.com icinga2[28183]: [2018-07-16 10:27:56 +0200] information/Application: Reload requested, lettin... over.
Jul 16 10:28:03 icinga2.vagrant.demo.icinga.com safe-reload[2450]: Validating config files: Done
Jul 16 10:28:03 icinga2.vagrant.demo.icinga.com safe-reload[2450]: Reloading Icinga 2: Done
Jul 16 10:28:03 icinga2.vagrant.demo.icinga.com systemd[1]: icinga2.service: Supervising process 2495 which is not our child. We'll most like...exits.
Jul 16 10:28:04 icinga2.vagrant.demo.icinga.com systemd[1]: Reloaded Icinga host/service/network monitoring system.
Hint: Some lines were ellipsized, use -l to show in full.
[root@icinga2 ~]#
[root@icinga2 ~]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.8.4-816-g8c8f1de)

Copyright (c) 2012-2018 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-693.17.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: unknown

Highly likely the deployed data really contains config validation errors, the debug log from systemd doesn't look suspicious.

Crunsher commented 6 years ago

I could not reproduce this either:

[root@localhost ~]# icinga2 object list --name foo
[root@localhost ~]# systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2018-07-16 10:47:50 CEST; 22s ago
  Process: 10645 ExecStartPre=/usr/lib/icinga2/prepare-dirs (code=exited, status=0/SUCCESS)
 Main PID: 10655 (icinga2)
   CGroup: /system.slice/icinga2.service
           ├─10655 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e
           └─10689 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e

Jul 16 10:47:50 localhost.localdomain systemd[1]: Starting Icinga host/service/network monitoring system...
Jul 16 10:47:50 localhost.localdomain systemd[1]: Started Icinga host/service/network monitoring system.
[root@localhost ~]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.8.4-816-g8c8f1de)

Copyright (c) 2012-2018 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-862.6.3.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: unknown
[root@localhost ~]# systemctl reload icinga2
[root@localhost ~]# icinga2 object list --name foo
Object 'foo' of type 'Host':
  % declared in '/etc/icinga2/icinga2.conf', lines 61:1-61:17
Mikesch-mp commented 6 years ago

If you have ipv6 i will give you access I dont get it, because if i do systemctl restart icinga2 new config gets loaded without any error.

dnsmichi commented 6 years ago

I don't have any IPv6 at hand, I would be interested in a reliable test case to reproduce the problem.

Exit code 7 leads me to the guess, that safe-reload does something suspicious here. The Icinga 2 PID-File doesn't exist at the point where it is expected, or similar.

Can you please share your current /etc/sysconfig/icinga2 configuration file? Furthermore, extract ICINGA2_PID_FILE and verify that the file is there and accessible.

If that's the case, check wether the PIDs written in there actually belong to the Icinga process currently running. If kill -HUP fails, you'll also end up with exit code 7 and no reload possible.

Mikesch-mp commented 6 years ago

Runnung safe-reload manualy bringst the expected result

[root@koelin01 system]# systemctl reload icinga2
Job for icinga2.service failed because the control process exited with error code. See "systemctl status icinga2.service" and "journalctl -xe" for details.
[root@koelin01 system]# /usr/lib/icinga2/safe-reload
Validating config files: Done
Reloading Icinga 2: Done
[root@koelin01 system]# echo $?
0

The syconfig file

[root@koelin01 system]# cat /etc/sysconfig/icinga2
DAEMON=/usr/sbin/icinga2
ICINGA2_CONFIG_FILE=/etc/icinga2/icinga2.conf
ICINGA2_RUN_DIR=/run
ICINGA2_STATE_DIR=/var
ICINGA2_PID_FILE=$ICINGA2_RUN_DIR/icinga2/icinga2.pid
ICINGA2_LOG_DIR=/var/log/icinga2
ICINGA2_ERROR_LOG=/var/log/icinga2/error.log
ICINGA2_STARTUP_LOG=/var/log/icinga2/startup.log
ICINGA2_LOG=/var/log/icinga2/icinga2.log
ICINGA2_CACHE_DIR=/var/cache/icinga2
ICINGA2_USER=icinga
ICINGA2_GROUP=icinga
ICINGA2_COMMAND_GROUP=icingacmd

The pid file

[root@koelin01 system]# ls -la /run/icinga2/icinga2.pid
-rw-r--r-- 1 icinga icinga 6 Jul 16 13:55 /run/icinga2/icinga2.pid

Wasnt there something with variables doesnt work in sysconfig files?

If i change the line for the pidfile to

ICINGA2_PID_FILE=/run/icinga2/icinga2.pid

Reload works perfect. I never touched the sysconfig file, only used from the rpm provided by the repository.

Mikesch-mp commented 6 years ago

The sysconfig file is only saved as icinga2.rpmnew when there is a file from an older version. Maybe add an hint to the rename the old file before upgrading?

dnsmichi commented 6 years ago

Ok, I see it. This is a long standing bug with sysconfig and path expansion based on shell variables. This only works in SysVinit, but not with Systemd.

a49a598eef etc/initsystem/icinga2.sysconfig.cmake (Gunnar Beutner 2014-07-22 13:18:41 +0200 5) ICINGA2_PID_FILE=$ICINGA2_RUN_DIR/icinga2/icinga2.pid

never resulted in the correct PID path and as such safe-reload would always fail and return 7. Maybe the file was written into that location.

https://github.com/systemd/systemd/issues/2123

Since you're using the old sysconfig file, and it doesn't get overridden on upgrade, we'll need to highlight this in the upgrading docs. Thanks for testing 👍

2.9 provides the correct environment variables for Systemd too, and uses sane default values. Basically you can just remove the content of the sysconfig file unless you've made specific changes to RLIMIT or user/groups.

Mikesch-mp commented 6 years ago

I removed it and everything runs like expected now :)