brudrafon commented 6 years ago

After Icinga2 safe-reload icinga2.service is killed in 30 minutes by systemd, new process starts immediately without any issue. During that time, new icinga2 process is running without any write to icinga2.log and alerts are send for service/hosts in downtime. Systemd reports icinga2 process as reloading. We start observing this issue after upgrade to 2.10.1 from 2.9.2

Logs - Systemd

Nov 05 13:22:27 icinga2-master01 safe-reload[22235]: Validating config files: Done Nov 05 13:22:27 icinga2-master01 safe-reload[22235]: Reloading Icinga 2: Done Nov 05 13:23:58 icinga2-master01 systemd[1]: icinga2.service: Supervising process 22938 which is not our child. We'll most likely not notice when it exits. Nov 05 13:53:58 icinga2-master01 systemd[1]: icinga2.service stop-sigterm timed out. Killing. Nov 05 13:53:58 icinga2-master01 systemd[1]: Unit icinga2.service entered failed state. Nov 05 13:53:58 icinga2-master01 systemd[1]: icinga2.service failed. Nov 05 13:53:59 icinga2-master01 systemd[1]: icinga2.service holdoff time over, scheduling restart. Nov 05 13:53:59 icinga2-master01 systemd[1]: Starting Icinga host/service/network monitoring system...

Logs - Icinga2

[2018-11-05 13:22:00 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2018-11-05 13:22:02 +0100] information/WorkQueue: #14 (JsonRpcConnection, #3) items: 1, rate: 338.8/s (20328/min 102301/5min 307400/15min);
[2018-11-05 13:22:16 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 10396, rate: 2563.1/s (153786/min 1112240/5min 3260604/15min); empty in 9 seconds
[2018-11-05 13:22:16 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 4, rate: 4697.67/s (281860/min 1415698/5min 4248099/15min);
[2018-11-05 13:22:26 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 13894, rate: 2500.98/s (150059/min 1053745/5min 3257222/15min); empty in 39 seconds

[2018-11-05 13:22:27 +0100] information/Application: Got reload command: Starting new instance.

[2018-11-05 13:22:28 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2, rate: 3955.5/s (237330/min 1188565/5min 3564845/15min);
[2018-11-05 13:22:36 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 15857, rate: 2604.85/s (156291/min 1051027/5min 3254771/15min); empty in 1 minute and 20 seconds
[2018-11-05 13:22:36 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 3, rate: 4764.02/s (285841/min 1415660/5min 4247138/15min);
[2018-11-05 13:22:46 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 19801, rate: 2485.17/s (149110/min 1047509/5min 3251191/15min); empty in 50 seconds
[2018-11-05 13:22:46 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 2, rate: 4715/s (282900/min 1415833/5min 4247791/15min);
[2018-11-05 13:22:48 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1, rate: 3959.95/s (237597/min 1188961/5min 3565043/15min);
[2018-11-05 13:22:56 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 17911, rate: 2460.82/s (147649/min 933246/5min 3253322/15min); empty in less than 1 millisecond
[2018-11-05 13:22:56 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 3, rate: 4722.85/s (283371/min 1416475/5min 4248406/15min);
[2018-11-05 13:23:06 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 11322, rate: 2535.43/s (152126/min 940128/5min 3262285/15min);
[2018-11-05 13:23:06 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 3, rate: 4728.52/s (283711/min 1416221/5min 4252361/15min);
[2018-11-05 13:23:16 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 6851, rate: 2806.55/s (168393/min 901298/5min 3264121/15min);
[2018-11-05 13:23:16 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 3, rate: 4726.22/s (283573/min 1418647/5min 4249986/15min);
[2018-11-05 13:23:18 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1, rate: 3948.92/s (236935/min 1189348/5min 3564738/15min);
[2018-11-05 13:23:26 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 4705, rate: 2862.48/s (171749/min 895520/5min 3269416/15min);
[2018-11-05 13:23:26 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 1, rate: 4700.35/s (282021/min 1415562/5min 4255922/15min);
[2018-11-05 13:23:36 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 8444, rate: 2855.58/s (171335/min 891030/5min 3261321/15min); empty in 22 seconds
[2018-11-05 13:23:36 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 3, rate: 4577.17/s (274630/min 1411150/5min 4239950/15min);
[2018-11-05 13:23:38 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 5, rate: 3931.17/s (235870/min 1189679/5min 3562370/15min);
[2018-11-05 13:23:46 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 9150, rate: 2819.58/s (169175/min 887733/5min 3259616/15min);
[2018-11-05 13:23:46 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 2, rate: 4542.57/s (272554/min 1405512/5min 4238478/15min);
[2018-11-05 13:23:48 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 3927.5/s (235650/min 1186940/5min 3563019/15min);
[2018-11-05 13:23:53 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 877.933/s (52676/min 264891/5min 794665/15min);
[2018-11-05 13:23:55 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-05 13:23:56 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 5074, rate: 2884.48/s (173069/min 817190/5min 3266160/15min);
[2018-11-05 13:23:56 +0100] information/WorkQueue: #11 (JsonRpcConnection, #0) items: 3, rate: 4617.73/s (277064/min 1410803/5min 4243321/15min);

[2018-11-05 13:23:58 +0100] information/Application: Reload requested, letting new process take over.
[2018-11-05 13:23:58 +0100] information/ApiListener: 'api' stopped.
[2018-11-05 13:24:00 +0100] information/ExternalCommandListener: 'command' stopped.

[2018-11-05 13:54:54 +0100] information/FileLogger: 'main-log' started.
[2018-11-05 13:55:01 +0100] information/ApiListener: 'api' started.

Service status in systemd

● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/icinga2.service.d
           └─autorestart.conf, coredump.conf
   Active: reloading (reload) since Fri 2018-11-02 15:14:06 CET; 2 days ago
  Process: 22235 ExecReload=/usr/lib/icinga2/safe-reload /etc/sysconfig/icinga2 (code=exited, status=0/SUCCESS)
  Process: 21862 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 22938
   CGroup: /system.slice/icinga2.service
           ├─27007 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /mnt/log/icinga2-error/error.log --reload-internal 31183
           └─27033 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /mnt/log/icinga2-error/error.log --reload-internal 31183

Expected Behavior

Icinga2 safe-reload is done in short time, icinga2 logs into log file, alerts are not send during reload.

Current Behavior

After icinga2 safe-reload, new icinga2 process is started, but no logs in icinga2/icinga.log and systemd kills icinga2.service after 30 minutes. Alerts are send for service/host in downtime during that period. Next started icinga2 process by systemd works as expected.

Possible Solution

Help us resolve strange race condition with icinga2 safe-reload and systemd.

Steps to Reproduce (for bugs)

Fails once/twice per day, not able to reproduce manually.

Context

During weird state of Icinga2 alerts are send for service/host with downtime set much longer before. Systemd reports icinga2.service as reloading

Your Environment

Version used (icinga2 --version):


icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.1-1)

Copyright (c) 2012-2018 Icinga Development Team (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

System information: Platform: CentOS Linux Platform version: 7 (Core) Kernel: Linux Kernel version: 3.10.0-862.9.1.el7.x86_64 Architecture: x86_64

Build information: Compiler: GNU 4.8.5 Build host: unknown

Application information:

General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2

Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var

Internal paths: Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid

* Operating System and version:

OS: Centos 7.5 Systemd version: systemd-219-57.el7_5.3.x86_64

* Enabled features (`icinga2 feature list`):

Disabled features: checker compatlog debuglog elasticsearch gelf influxdb livestatus opentsdb perfdata statusdata syslog Enabled features: api command graphite ido-mysql mainlog notification

* Icinga Web 2 version and modules (System - About):

Icinga Web 2 Version 2.6.1 Git commit b3e0b5d587c3587a09d82b3634dbe3a5dd315353

* Config validation (`icinga2 daemon -C`):

[2018-11-05 18:36:56 +0100] information/cli: Icinga application loader (version: r2.10.1-1) [2018-11-05 18:36:56 +0100] information/cli: Loading configuration file(s). [2018-11-05 18:36:56 +0100] information/ConfigItem: Committing config item(s). [2018-11-05 18:36:56 +0100] information/ApiListener: My API identity: icinga2-master01.change.me [2018-11-05 18:37:06 +0100] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 0, rate: 8.8/s (528/min 528/5min 528/15min); [2018-11-05 18:37:06 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-05 18:37:06 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-05 18:37:06 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-05 18:37:06 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 0, rate: 0/s (0/min 0/5min 0/15min);

[2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 35 ScheduledDowntimes. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 69225 Services. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 1 IcingaApplication. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 2489 Hosts. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 1 FileLogger. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 66299 Dependencies. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 4 NotificationCommands. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 131137 Notifications. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 1 NotificationComponent. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 105 HostGroups. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 1 ApiListener. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 406 Downtimes. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 1 GraphiteWriter. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 39 Comments. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 8 Zones. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 11 Endpoints. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 2 ApiUsers. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 33 Users. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 80 CheckCommands. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 18 UserGroups. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 4 ServiceGroups. [2018-11-05 18:38:35 +0100] information/ConfigItem: Instantiated 2 TimePeriods. [2018-11-05 18:38:35 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2018-11-05 18:38:35 +0100] information/cli: Finished validating the configuration file(s).

mcktr commented 6 years ago

After icinga2 safe-reload

What do you mean with that? How do you reload/restart the Icinga daemon?

brudrafon commented 6 years ago

After icinga2 safe-reload

What do you mean with that? How do you reload/restart the Icinga daemon?

We run systemctl reload icinga2

Which is mapped in systemd to /usr/lib/icinga2/safe-reload

cat /usr/lib/systemd/system/icinga2.service 
[Unit]
Description=Icinga host/service/network monitoring system
After=syslog.target network-online.target postgresql.service mariadb.service carbon-cache.service carbon-relay.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/icinga2
ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2
ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG}
PIDFile=/run/icinga2/icinga2.pid
ExecReload=/usr/lib/icinga2/safe-reload /etc/sysconfig/icinga2
TimeoutStartSec=30m

# Systemd >228 enforces a lower process number for services.
# Depending on the distribution and Systemd version, this must
# be explicitly raised. Packages will set the needed values
# into /etc/systemd/system/icinga2.service.d/limits.conf
#
# Please check the troubleshooting documentation for further details.
# The values below can be used as examples for customized service files.

#TasksMax=infinity
#LimitNPROC=62883

[Install]
WantedBy=multi-user.target

mcktr commented 6 years ago

I was able to reproduce this in the vagrant standalone box with the v2.10.1-1 stable packages and a high load on the system. With a normal load I was not able to reproduce this though.

const countHosts = 5000;
const countServices = 3;

for (i in range(countHosts)) {
  object Host "many-test-" + i use (i) {
    check_command = "random"
    address = "127.0.0.1"

  }
}

for (j in range(countServices)) {
  apply Service "many-test-" + j {
    check_command = "random"

    check_interval = 30s
    retry_interval = 30s

    assign where match("many*", host.name)
  }
}

Nov 08 19:04:06 icinga2.vagrant.demo.icinga.com systemd[1]: Started Icinga host/service/network monitoring system.
Nov 08 19:09:07 icinga2.vagrant.demo.icinga.com safe-reload[12143]: Validating config files: Done
Nov 08 19:09:07 icinga2.vagrant.demo.icinga.com safe-reload[12143]: Reloading Icinga 2: Done
Nov 08 19:09:07 icinga2.vagrant.demo.icinga.com systemd[1]: Reloaded Icinga host/service/network monitoring system.
Nov 08 19:09:15 icinga2.vagrant.demo.icinga.com systemd[1]: icinga2.service: Supervising process 12949 which is not our child. We'll most likely not notice when it exits.
Nov 08 19:39:08 icinga2.vagrant.demo.icinga.com systemd[1]: icinga2.service reload operation timed out. Stopping.
Nov 08 19:40:38 icinga2.vagrant.demo.icinga.com systemd[1]: icinga2.service stop-sigterm timed out. Killing.
Nov 08 19:40:38 icinga2.vagrant.demo.icinga.com systemd[1]: Unit icinga2.service entered failed state.
Nov 08 19:40:38 icinga2.vagrant.demo.icinga.com systemd[1]: icinga2.service failed.

I guess this has something to do with the IDO load, since I noticed that it happens when the WorkQueue for the IDO connection begins to grow.

[2018-11-08 19:04:23 +0100] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 168571, rate: 1.4/s (84/min 84/5min 84/15min); empty in 18 seconds

In the next step I'll try to reproduce this in the snapshot packages.

dnsmichi commented 6 years ago

I would first analyse why the IDO queries are not processed this fast. This is blocking the final shutdown request.

mcktr commented 6 years ago

I tried to analyse this a bit further.

Analysis

In the log we see that it stops at stopping the components.

[2018-11-11 17:23:09 +0100] information/Application: Got reload command: Starting new instance.
[2018-11-11 17:23:10 +0100] information/Application: Reload requested, letting new process take over.
[2018-11-11 17:23:10 +0100] information/ApiListener: 'api' stopped.
[2018-11-11 17:23:10 +0100] information/CheckerComponent: 'checker' stopped.
[2018-11-11 17:23:10 +0100] information/ExternalCommandListener: 'command' stopped.

While running the daemon in foreground I observed that there are log entries that are not logged to the file.

[2018-11-11 17:23:09 +0100] information/Application: Got reload command: Starting new instance.
[2018-11-11 17:23:10 +0100] information/Application: Reload requested, letting new process take over.
[2018-11-11 17:23:10 +0100] information/ApiListener: 'api' stopped.
[2018-11-11 17:23:10 +0100] information/CheckerComponent: 'checker' stopped.
[2018-11-11 17:23:10 +0100] information/ExternalCommandListener: 'command' stopped.
[2018-11-11 17:23:10 +0100] information/GraphiteWriter: 'graphite' paused.
[2018-11-11 17:23:10 +0100] information/WorkQueue: #6 (IdoMysqlConnection, ido-mysql) items: 102729, rate: 1276.7/s (76602/min 76602/5min 76602/15min); empty in 51 seconds
[2018-11-11 17:23:11 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 567, rate: 804.2/s (48252/min 48252/5min 48252/15min);
[2018-11-11 17:23:20 +0100] information/WorkQueue: #6 (IdoMysqlConnection, ido-mysql) items: 26599, rate: 2545.98/s (152759/min 152759/5min 152759/15min);
[2018-11-11 17:23:21 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 568, rate: 804.2/s (48252/min 48252/5min 48252/15min); empty in 1 hour, 34 minutes and 39 seconds
[2018-11-11 17:23:25 +0100] information/IdoMysqlConnection: Finished reconnecting to MySQL IDO database in 44.0784 second(s).
[2018-11-11 17:23:31 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 569, rate: 804.2/s (48252/min 48252/5min 48252/15min); empty in 1 hour, 34 minutes and 49 seconds
[2018-11-11 17:23:41 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 570, rate: 768.55/s (46113/min 48252/5min 48252/15min); empty in 1 hour, 34 minutes and 59 seconds
[2018-11-11 17:23:51 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 571, rate: 449.8/s (26988/min 48252/5min 48252/15min); empty in 1 hour, 35 minutes and 9 seconds
[2018-11-11 17:24:01 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 572, rate: 159.683/s (9581/min 48252/5min 48252/15min); empty in 1 hour, 35 minutes and 19 seconds

This is possibly caused by the filelogger is stopped before all other components are fully stopped.

Having a look with gdb to find out what's going on during the reload process and why it's never finish.

Thread 102 (Thread 0x7ffff1d59700 (LWP 19074)):
#0  0x00007ffff507ebf9 in __GI___poll (fds=0x7fffbe103f00, nfds=1, timeout=500) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00000000013fd877 in icinga::Process::IOThreadProc (tid=3) at /home/michael/Coding/Icinga/icinga2/lib/base/process.cpp:688
#2  0x00000000014054c1 in std::__invoke_impl<void, void (*&)(int), int&> (__f=@0x2543968: 0x13fd540 <icinga::Process::IOThreadProc(int)>, __args=@0x2543970: 3)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#3  0x0000000001405462 in std::__invoke<void (*&)(int), int&> (__fn=@0x2543968: 0x13fd540 <icinga::Process::IOThreadProc(int)>, __args=@0x2543970: 3)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#4  0x0000000001405420 in std::_Bind<void (*(int))(int)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x2543968, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#5  0x00000000014053d6 in std::_Bind<void (*(int))(int)>::operator()<, void>() (this=0x2543968) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#6  0x000000000140539d in std::__invoke_impl<void, std::_Bind<void (*(int))(int)>>(std::__invoke_other, std::_Bind<void (*(int))(int)>&&) (__f=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#7  0x000000000140532d in std::__invoke<std::_Bind<void (*(int))(int)>>(std::_Bind<void (*(int))(int)>&&) (__fn=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#8  0x0000000001405305 in std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > >::_M_invoke<0ul> (this=0x2543968)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:234
#9  0x00000000014052d5 in std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > >::operator()() (this=0x2543968)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:243
#10 0x0000000001405089 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > > >::_M_run() (this=0x2543960)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:186
#11 0x00007ffff59ce57f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ffff69ee6db in start_thread (arg=0x7ffff1d59700) at pthread_create.c:463
#13 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 101 (Thread 0x7ffff7e7f700 (LWP 19073)):
#0  0x00007ffff507ebf9 in __GI___poll (fds=0x7fffd068e5e0, nfds=1, timeout=500) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00000000013fd877 in icinga::Process::IOThreadProc (tid=2) at /home/michael/Coding/Icinga/icinga2/lib/base/process.cpp:688
#2  0x00000000014054c1 in std::__invoke_impl<void, void (*&)(int), int&> (__f=@0x2543a08: 0x13fd540 <icinga::Process::IOThreadProc(int)>, __args=@0x2543a10: 2)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#3  0x0000000001405462 in std::__invoke<void (*&)(int), int&> (__fn=@0x2543a08: 0x13fd540 <icinga::Process::IOThreadProc(int)>, __args=@0x2543a10: 2)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#4  0x0000000001405420 in std::_Bind<void (*(int))(int)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x2543a08, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#5  0x00000000014053d6 in std::_Bind<void (*(int))(int)>::operator()<, void>() (this=0x2543a08) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#6  0x000000000140539d in std::__invoke_impl<void, std::_Bind<void (*(int))(int)>>(std::__invoke_other, std::_Bind<void (*(int))(int)>&&) (__f=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#7  0x000000000140532d in std::__invoke<std::_Bind<void (*(int))(int)>>(std::_Bind<void (*(int))(int)>&&) (__fn=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#8  0x0000000001405305 in std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > >::_M_invoke<0ul> (this=0x2543a08)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:234
#9  0x00000000014052d5 in std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > >::operator()() (this=0x2543a08)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:243
#10 0x0000000001405089 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > > >::_M_run() (this=0x2543a00)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:186
#11 0x00007ffff59ce57f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ffff69ee6db in start_thread (arg=0x7ffff7e7f700) at pthread_create.c:463
#13 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 100 (Thread 0x7ffff7e3e700 (LWP 19072)):
#0  0x00007ffff507ebf9 in __GI___poll (fds=0x7fffdd5c6cb0, nfds=1, timeout=500) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00000000013fd877 in icinga::Process::IOThreadProc (tid=1) at /home/michael/Coding/Icinga/icinga2/lib/base/process.cpp:688
#2  0x00000000014054c1 in std::__invoke_impl<void, void (*&)(int), int&> (__f=@0x2543928: 0x13fd540 <icinga::Process::IOThreadProc(int)>, __args=@0x2543930: 1)
---Type <return> to continue, or q <return> to quit---
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#3  0x0000000001405462 in std::__invoke<void (*&)(int), int&> (__fn=@0x2543928: 0x13fd540 <icinga::Process::IOThreadProc(int)>, __args=@0x2543930: 1)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#4  0x0000000001405420 in std::_Bind<void (*(int))(int)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x2543928, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#5  0x00000000014053d6 in std::_Bind<void (*(int))(int)>::operator()<, void>() (this=0x2543928) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#6  0x000000000140539d in std::__invoke_impl<void, std::_Bind<void (*(int))(int)>>(std::__invoke_other, std::_Bind<void (*(int))(int)>&&) (__f=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#7  0x000000000140532d in std::__invoke<std::_Bind<void (*(int))(int)>>(std::_Bind<void (*(int))(int)>&&) (__fn=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#8  0x0000000001405305 in std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > >::_M_invoke<0ul> (this=0x2543928)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:234
#9  0x00000000014052d5 in std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > >::operator()() (this=0x2543928)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:243
#10 0x0000000001405089 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > > >::_M_run() (this=0x2543920)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:186
#11 0x00007ffff59ce57f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ffff69ee6db in start_thread (arg=0x7ffff7e3e700) at pthread_create.c:463
#13 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 99 (Thread 0x7ffff12a4700 (LWP 19071)):
#0  0x00007ffff507ebf9 in __GI___poll (fds=0x7fffe6a5c8b0, nfds=2, timeout=289793) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00000000013fd877 in icinga::Process::IOThreadProc (tid=0) at /home/michael/Coding/Icinga/icinga2/lib/base/process.cpp:688
#2  0x00000000014054c1 in std::__invoke_impl<void, void (*&)(int), int&> (__f=@0x2543948: 0x13fd540 <icinga::Process::IOThreadProc(int)>, __args=@0x2543950: 0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#3  0x0000000001405462 in std::__invoke<void (*&)(int), int&> (__fn=@0x2543948: 0x13fd540 <icinga::Process::IOThreadProc(int)>, __args=@0x2543950: 0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#4  0x0000000001405420 in std::_Bind<void (*(int))(int)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x2543948, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#5  0x00000000014053d6 in std::_Bind<void (*(int))(int)>::operator()<, void>() (this=0x2543948) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#6  0x000000000140539d in std::__invoke_impl<void, std::_Bind<void (*(int))(int)>>(std::__invoke_other, std::_Bind<void (*(int))(int)>&&) (__f=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#7  0x000000000140532d in std::__invoke<std::_Bind<void (*(int))(int)>>(std::_Bind<void (*(int))(int)>&&) (__fn=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#8  0x0000000001405305 in std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > >::_M_invoke<0ul> (this=0x2543948)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:234
#9  0x00000000014052d5 in std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > >::operator()() (this=0x2543948)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:243
#10 0x0000000001405089 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (*(int))(int)> > > >::_M_run() (this=0x2543940)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:186
#11 0x00007ffff59ce57f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ffff69ee6db in start_thread (arg=0x7ffff12a4700) at pthread_create.c:463
#13 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 72 (Thread 0x7ffff0f98700 (LWP 18970)):
#0  0x00007ffff69f49f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24b3638 <icinga::Application::GetTP()::tp+3112>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x24b35e8 <icinga::Application::GetTP()::tp+3032>, cond=0x24b3610 <icinga::Application::GetTP()::tp+3072>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x24b3610 <icinga::Application::GetTP()::tp+3072>, mutex=0x24b35e8 <icinga::Application::GetTP()::tp+3032>) at pthread_cond_wait.c:655
#3  0x0000000001436df3 in boost::condition_variable::wait (this=0x24b35e8 <icinga::Application::GetTP()::tp+3032>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:81
#4  0x000000000146d81a in icinga::ThreadPool::WorkerThread::ThreadProc (this=0x24b3780 <icinga::Application::GetTP()::tp+3440>, queue=...) at /home/michael/Coding/Icinga/icinga2/lib/base/threadpool.cpp:110
#5  0x00000000014736c6 in std::__invoke_impl<void, void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&> (__f=
    @0x7fffd8004658: (void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::WorkerThread * const, icinga::ThreadPool::Queue &)) 0x146d590 <icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&)>, __t=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:66
---Type <return> to continue, or q <return> to quit---
#6  0x0000000001473597 in std::__invoke<void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&> (__fn=
    @0x7fffd8004658: (void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::WorkerThread * const, icinga::ThreadPool::Queue &)) 0x146d590 <icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&)>, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#7  0x0000000001473548 in std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7fffd8004658, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#8  0x00000000014734d6 in std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)>::operator()<, void>() (this=0x7fffd8004658) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#9  0x0000000001472f7c in boost::detail::thread_data<std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)> >::run() (this=0x7fffd80044a0) at /usr/include/boost/thread/detail/thread.hpp:116
#10 0x00007ffff79bdbcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#11 0x00007ffff69ee6db in start_thread (arg=0x7ffff0f98700) at pthread_create.c:463
#12 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 70 (Thread 0x7ffff101a700 (LWP 18968)):
#0  0x00007ffff69f49f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24b32f8 <icinga::Application::GetTP()::tp+2280>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x24b32a8 <icinga::Application::GetTP()::tp+2200>, cond=0x24b32d0 <icinga::Application::GetTP()::tp+2240>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x24b32d0 <icinga::Application::GetTP()::tp+2240>, mutex=0x24b32a8 <icinga::Application::GetTP()::tp+2200>) at pthread_cond_wait.c:655
#3  0x0000000001436df3 in boost::condition_variable::wait (this=0x24b32a8 <icinga::Application::GetTP()::tp+2200>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:81
#4  0x000000000146d81a in icinga::ThreadPool::WorkerThread::ThreadProc (this=0x24b3440 <icinga::Application::GetTP()::tp+2608>, queue=...) at /home/michael/Coding/Icinga/icinga2/lib/base/threadpool.cpp:110
#5  0x00000000014736c6 in std::__invoke_impl<void, void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&> (__f=
    @0x7fffd8003f58: (void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::WorkerThread * const, icinga::ThreadPool::Queue &)) 0x146d590 <icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&)>, __t=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:66
#6  0x0000000001473597 in std::__invoke<void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&> (__fn=
    @0x7fffd8003f58: (void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::WorkerThread * const, icinga::ThreadPool::Queue &)) 0x146d590 <icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&)>, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#7  0x0000000001473548 in std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7fffd8003f58, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#8  0x00000000014734d6 in std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)>::operator()<, void>() (this=0x7fffd8003f58) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#9  0x0000000001472f7c in boost::detail::thread_data<std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)> >::run() (this=0x7fffd8003da0) at /usr/include/boost/thread/detail/thread.hpp:116
#10 0x00007ffff79bdbcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#11 0x00007ffff69ee6db in start_thread (arg=0x7ffff101a700) at pthread_create.c:463
#12 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 68 (Thread 0x7ffff109c700 (LWP 18966)):
#0  0x00007ffff69f49f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24b2fbc <icinga::Application::GetTP()::tp+1452>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x24b2f68 <icinga::Application::GetTP()::tp+1368>, cond=0x24b2f90 <icinga::Application::GetTP()::tp+1408>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x24b2f90 <icinga::Application::GetTP()::tp+1408>, mutex=0x24b2f68 <icinga::Application::GetTP()::tp+1368>) at pthread_cond_wait.c:655
#3  0x0000000001436df3 in boost::condition_variable::wait (this=0x24b2f68 <icinga::Application::GetTP()::tp+1368>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:81
#4  0x000000000146d81a in icinga::ThreadPool::WorkerThread::ThreadProc (this=0x24b3100 <icinga::Application::GetTP()::tp+1776>, queue=...) at /home/michael/Coding/Icinga/icinga2/lib/base/threadpool.cpp:110
#5  0x00000000014736c6 in std::__invoke_impl<void, void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&> (__f=
    @0x7fffd8003858: (void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::WorkerThread * const, icinga::ThreadPool::Queue &)) 0x146d590 <icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&)>, __t=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:66
#6  0x0000000001473597 in std::__invoke<void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&> (__fn=
    @0x7fffd8003858: (void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::WorkerThread * const, icinga::ThreadPool::Queue &)) 0x146d590 <icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&)>, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#7  0x0000000001473548 in std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7fffd8003858, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#8  0x00000000014734d6 in std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)>::operator()<, void>() (this=0x7fffd8003858) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
---Type <return> to continue, or q <return> to quit---
#9  0x0000000001472f7c in boost::detail::thread_data<std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)> >::run() (this=0x7fffd80036a0) at /usr/include/boost/thread/detail/thread.hpp:116
#10 0x00007ffff79bdbcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#11 0x00007ffff69ee6db in start_thread (arg=0x7ffff109c700) at pthread_create.c:463
#12 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 66 (Thread 0x7ffff111e700 (LWP 18964)):
#0  0x00007ffff69f49f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x24b2c78 <icinga::Application::GetTP()::tp+616>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x24b2c28 <icinga::Application::GetTP()::tp+536>, cond=0x24b2c50 <icinga::Application::GetTP()::tp+576>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x24b2c50 <icinga::Application::GetTP()::tp+576>, mutex=0x24b2c28 <icinga::Application::GetTP()::tp+536>) at pthread_cond_wait.c:655
#3  0x0000000001436df3 in boost::condition_variable::wait (this=0x24b2c28 <icinga::Application::GetTP()::tp+536>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:81
#4  0x000000000146d81a in icinga::ThreadPool::WorkerThread::ThreadProc (this=0x24b2dc0 <icinga::Application::GetTP()::tp+944>, queue=...) at /home/michael/Coding/Icinga/icinga2/lib/base/threadpool.cpp:110
#5  0x00000000014736c6 in std::__invoke_impl<void, void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&> (__f=
    @0x7fffd8003158: (void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::WorkerThread * const, icinga::ThreadPool::Queue &)) 0x146d590 <icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&)>, __t=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:66
#6  0x0000000001473597 in std::__invoke<void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&> (__fn=
    @0x7fffd8003158: (void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::WorkerThread * const, icinga::ThreadPool::Queue &)) 0x146d590 <icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&)>, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#7  0x0000000001473548 in std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7fffd8003158, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#8  0x00000000014734d6 in std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)>::operator()<, void>() (this=0x7fffd8003158) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#9  0x0000000001472f7c in boost::detail::thread_data<std::_Bind<void (icinga::ThreadPool::WorkerThread::*(std::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::reference_wrapper<icinga::ThreadPool::Queue>))(icinga::ThreadPool::Queue&)> >::run() (this=0x7fffd8002fa0) at /usr/include/boost/thread/detail/thread.hpp:116
#10 0x00007ffff79bdbcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#11 0x00007ffff69ee6db in start_thread (arg=0x7ffff111e700) at pthread_create.c:463
#12 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 56 (Thread 0x7ffff1ddb700 (LWP 18950)):
#0  0x00007ffff69f882e in __libc_recv (fd=19, buf=0x7fffc0276730, len=16384, flags=0) at ../sysdeps/unix/sysv/linux/recv.c:28
#1  0x00007ffff1560ca3 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#2  0x00007ffff1560d25 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#3  0x00007ffff153ab3e in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#4  0x00007ffff153bbbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#5  0x00007ffff152fe6c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#6  0x00007ffff1530f38 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#7  0x00007ffff152d6ce in mysql_next_result () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

#8  0x00007ffff1b16d89 in MysqlInterfaceImpl::next_result (this=0x7fffe0007440, mysql=0x7fffe0001828) at /home/michael/Coding/Icinga/icinga2/lib/mysql_shim/mysqlinterface.cpp:88
#9  0x0000000001a636de in icinga::IdoMysqlConnection::FinishAsyncQueries (this=0x7fffe0000fa0) at /home/michael/Coding/Icinga/icinga2/lib/db_ido_mysql/idomysqlconnection.cpp:585
#10 0x0000000001a7fa61 in std::__invoke_impl<void, void (icinga::IdoMysqlConnection::*&)(), icinga::IdoMysqlConnection*&> (
    __f=@0x7fffcc002ef0: (void (icinga::IdoMysqlConnection::*)(icinga::IdoMysqlConnection * const)) 0x1a627e0 <icinga::IdoMysqlConnection::FinishAsyncQueries()>, __t=@0x7fffcc002f00: 0x7fffe0000fa0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:73
#11 0x0000000001a7f992 in std::__invoke<void (icinga::IdoMysqlConnection::*&)(), icinga::IdoMysqlConnection*&> (
    __fn=@0x7fffcc002ef0: (void (icinga::IdoMysqlConnection::*)(icinga::IdoMysqlConnection * const)) 0x1a627e0 <icinga::IdoMysqlConnection::FinishAsyncQueries()>, __args=@0x7fffcc002f00: 0x7fffe0000fa0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#12 0x0000000001a80190 in std::_Bind<void (icinga::IdoMysqlConnection::*(icinga::IdoMysqlConnection*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7fffcc002ef0, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#13 0x0000000001a80126 in std::_Bind<void (icinga::IdoMysqlConnection::*(icinga::IdoMysqlConnection*))()>::operator()<, void>() (this=0x7fffcc002ef0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#14 0x0000000001a7fe7d in std::_Function_handler<void (), std::_Bind<void (icinga::IdoMysqlConnection::*(icinga::IdoMysqlConnection*))()> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/std_function.h:316
---Type <return> to continue, or q <return> to quit---

#15 0x00000000013be92e in std::function<void ()>::operator()() const (this=0x7ffff1ddab98) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/std_function.h:706
#16 0x00000000014be4e4 in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) (this=0x7fffe0001448, func=...) at /home/michael/Coding/Icinga/icinga2/lib/base/workqueue.cpp:253
#17 0x00000000014bdd86 in icinga::WorkQueue::WorkerThreadProc (this=0x7fffe0001448) at /home/michael/Coding/Icinga/icinga2/lib/base/workqueue.cpp:296
#18 0x00000000014c12e1 in std::__invoke_impl<void, void (icinga::WorkQueue::*&)(), icinga::WorkQueue*&> (
    __f=@0x7fffe4001438: (void (icinga::WorkQueue::*)(icinga::WorkQueue * const)) 0x14bda50 <icinga::WorkQueue::WorkerThreadProc()>, __t=@0x7fffe4001448: 0x7fffe0001448)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:73
#19 0x00000000014c1212 in std::__invoke<void (icinga::WorkQueue::*&)(), icinga::WorkQueue*&> (
    __fn=@0x7fffe4001438: (void (icinga::WorkQueue::*)(icinga::WorkQueue * const)) 0x14bda50 <icinga::WorkQueue::WorkerThreadProc()>, __args=@0x7fffe4001448: 0x7fffe0001448)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#20 0x00000000014c1a20 in std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7fffe4001438, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#21 0x00000000014c19d6 in std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()>::operator()<, void>() (this=0x7fffe4001438)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#22 0x00000000014c190c in boost::detail::thread_data<std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()> >::run() (this=0x7fffe4001280) at /usr/include/boost/thread/detail/thread.hpp:116
#23 0x00007ffff79bdbcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#24 0x00007ffff69ee6db in start_thread (arg=0x7ffff1ddb700) at pthread_create.c:463
#25 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 55 (Thread 0x7ffff1e1c700 (LWP 18949)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff69f1098 in __GI___pthread_mutex_lock (mutex=0x2c3ab20) at ../nptl/pthread_mutex_lock.c:113
#2  0x00000000013e7455 in boost::recursive_mutex::lock (this=0x2c3ab20) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:113
#3  0x00000000013e736b in icinga::ObjectLock::LockMutex (object=0x7fffe80021e0) at /home/michael/Coding/Icinga/icinga2/lib/base/objectlock.cpp:62
#4  0x00000000013e724e in icinga::ObjectLock::Lock (this=0x7ffff1e1aab0) at /home/michael/Coding/Icinga/icinga2/lib/base/objectlock.cpp:88
#5  0x00000000013e72d6 in icinga::ObjectLock::ObjectLock (this=0x7ffff1e1aab0, object=0x7fffe80021e0) at /home/michael/Coding/Icinga/icinga2/lib/base/objectlock.cpp:44

#6  0x0000000001b9458a in icinga::GraphiteWriter::SendMetric (this=0x7fffe80021e0, prefix=..., name=..., value=73.799999999999997, ts=1541953843.286433)
    at /home/michael/Coding/Icinga/icinga2/lib/perfdata/graphitewriter.cpp:302
#7  0x0000000001b94e12 in icinga::GraphiteWriter::SendPerfdata (this=0x7fffe80021e0, prefix=..., cr=..., ts=1541953843.286433) at /home/michael/Coding/Icinga/icinga2/lib/perfdata/graphitewriter.cpp:275
#8  0x0000000001b93e23 in icinga::GraphiteWriter::CheckResultHandlerInternal (this=0x7fffe80021e0, checkable=..., cr=...) at /home/michael/Coding/Icinga/icinga2/lib/perfdata/graphitewriter.cpp:247
#9  0x0000000001ba56fe in std::__invoke_impl<void, void (icinga::GraphiteWriter::*&)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&), icinga::GraphiteWriter*&, boost::intrusive_ptr<icinga::Checkable>&, boost::intrusive_ptr<icinga::CheckResult>&> (__f=
    @0x7fff3f4a4ac0: (void (icinga::GraphiteWriter::*)(icinga::GraphiteWriter * const, const boost::intrusive_ptr<icinga::Checkable> &, const boost::intrusive_ptr<icinga::CheckResult> &)) 0x1b92fe0 <icinga::GraphiteWriter::CheckResultHandlerInternal(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&)>, __t=@0x7fff3f4a4ae0: 0x7fffe80021e0, __args=..., __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:73
#10 0x0000000001ba55ec in std::__invoke<void (icinga::GraphiteWriter::*&)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&), icinga::GraphiteWriter*&, boost::intrusive_ptr<icinga::Checkable>&, boost::intrusive_ptr<icinga::CheckResult>&> (__fn=
    @0x7fff3f4a4ac0: (void (icinga::GraphiteWriter::*)(icinga::GraphiteWriter * const, const boost::intrusive_ptr<icinga::Checkable> &, const boost::intrusive_ptr<icinga::CheckResult> &)) 0x1b92fe0 <icinga::GraphiteWriter::CheckResultHandlerInternal(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&)>, __args=..., __args=..., __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#11 0x0000000001ba5580 in std::_Bind<void (icinga::GraphiteWriter::*(icinga::GraphiteWriter*, boost::intrusive_ptr<icinga::Checkable>, boost::intrusive_ptr<icinga::CheckResult>))(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&)>::__call<void, , 0ul, 1ul, 2ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul>) (this=0x7fff3f4a4ac0, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#12 0x0000000001ba54c6 in std::_Bind<void (icinga::GraphiteWriter::*(icinga::GraphiteWriter*, boost::intrusive_ptr<icinga::Checkable>, boost::intrusive_ptr<icinga::CheckResult>))(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&)>::operator()<, void>() (this=0x7fff3f4a4ac0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#13 0x0000000001ba519d in std::_Function_handler<void (), std::_Bind<void (icinga::GraphiteWriter::*(icinga::GraphiteWriter*, boost::intrusive_ptr<icinga::Checkable>, boost::intrusive_ptr<icinga::CheckResult>))(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&)> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/std_function.h:316
#14 0x00000000013be92e in std::function<void ()>::operator()() const (this=0x7ffff1e1bb98) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/std_function.h:706
#15 0x00000000014be4e4 in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) (this=0x7fffe8002380, func=...) at /home/michael/Coding/Icinga/icinga2/lib/base/workqueue.cpp:253
#16 0x00000000014bdd86 in icinga::WorkQueue::WorkerThreadProc (this=0x7fffe8002380) at /home/michael/Coding/Icinga/icinga2/lib/base/workqueue.cpp:296
#17 0x00000000014c12e1 in std::__invoke_impl<void, void (icinga::WorkQueue::*&)(), icinga::WorkQueue*&> (
---Type <return> to continue, or q <return> to quit---
    __f=@0x7fffd4001318: (void (icinga::WorkQueue::*)(icinga::WorkQueue * const)) 0x14bda50 <icinga::WorkQueue::WorkerThreadProc()>, __t=@0x7fffd4001328: 0x7fffe8002380)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:73
#18 0x00000000014c1212 in std::__invoke<void (icinga::WorkQueue::*&)(), icinga::WorkQueue*&> (
    __fn=@0x7fffd4001318: (void (icinga::WorkQueue::*)(icinga::WorkQueue * const)) 0x14bda50 <icinga::WorkQueue::WorkerThreadProc()>, __args=@0x7fffd4001328: 0x7fffe8002380)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#19 0x00000000014c1a20 in std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x7fffd4001318, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#20 0x00000000014c19d6 in std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()>::operator()<, void>() (this=0x7fffd4001318)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#21 0x00000000014c190c in boost::detail::thread_data<std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()> >::run() (this=0x7fffd4001160) at /usr/include/boost/thread/detail/thread.hpp:116
#22 0x00007ffff79bdbcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#23 0x00007ffff69ee6db in start_thread (arg=0x7ffff1e1c700) at pthread_create.c:463
#24 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 53 (Thread 0x7ffff7fc4700 (LWP 18947)):
#0  0x00007ffff507ebf9 in __GI___poll (fds=0x7ffff7fc0c38, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00000000014264c0 in icinga::Socket::Poll (this=0x7fffc40922e0, read=true, write=false, timeout=0x0) at /home/michael/Coding/Icinga/icinga2/lib/base/socket.cpp:417

#2  0x0000000001995a39 in icinga::ExternalCommandListener::CommandPipeThread (this=0x7fffbc000f40, commandPath=...) at /home/michael/Coding/Icinga/icinga2/lib/compat/externalcommandlistener.cpp:124
#3  0x00000000019a1d99 in std::__invoke_impl<void, void (icinga::ExternalCommandListener::*&)(icinga::String const&), icinga::ExternalCommandListener*&, icinga::String&> (__f=
    @0x2c79068: (void (icinga::ExternalCommandListener::*)(icinga::ExternalCommandListener * const, const icinga::String &)) 0x1995140 <icinga::ExternalCommandListener::CommandPipeThread(icinga::String const&)>, __t=@0x2c79098: 0x7fffbc000f40, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:73
#4  0x00000000019a1c97 in std::__invoke<void (icinga::ExternalCommandListener::*&)(icinga::String const&), icinga::ExternalCommandListener*&, icinga::String&> (__fn=
    @0x2c79068: (void (icinga::ExternalCommandListener::*)(icinga::ExternalCommandListener * const, const icinga::String &)) 0x1995140 <icinga::ExternalCommandListener::CommandPipeThread(icinga::String const&)>, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#5  0x00000000019a1c48 in std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x2c79068, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#6  0x00000000019a1bd6 in std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)>::operator()<, void>() (this=0x2c79068)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#7  0x00000000019a1b9d in std::__invoke_impl<void, std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)>>(std::__invoke_other, std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)>&&) (__f=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#8  0x00000000019a1b2d in std::__invoke<std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)>>(std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)>&&) (__fn=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#9  0x00000000019a1b05 in std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)> > >::_M_invoke<0ul> (
    this=0x2c79068) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:234
#10 0x00000000019a1ad5 in std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)> > >::operator()() (
    this=0x2c79068) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:243
#11 0x00000000019a1859 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ExternalCommandListener::*(icinga::ExternalCommandListener*, icinga::String))(icinga::String const&)> > > >::_M_run() (this=0x2c79060) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:186
#12 0x00007ffff59ce57f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff69ee6db in start_thread (arg=0x7ffff7fc4700) at pthread_create.c:463
#14 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 52 (Thread 0x7ffff7f83700 (LWP 18946)):
#0  0x00007ffff69f86d7 in __libc_accept (fd=15, addr=..., len=0x7ffff7f82934) at ../sysdeps/unix/sysv/linux/accept.c:26

#1  0x00000000014260bb in icinga::Socket::Accept (this=0x2c34ff0) at /home/michael/Coding/Icinga/icinga2/lib/base/socket.cpp:358
#2  0x000000000156ee4e in icinga::ApiListener::ListenerThreadProc (this=0x7fffc0046f40, server=...) at /home/michael/Coding/Icinga/icinga2/lib/remote/apilistener.cpp:382
#3  0x000000000159843a in std::__invoke_impl<void, void (icinga::ApiListener::*&)(boost::intrusive_ptr<icinga::Socket> const&), icinga::ApiListener*&, boost::intrusive_ptr<icinga::TcpSocket>&> (__f=
    @0x2a195d8: (void (icinga::ApiListener::*)(icinga::ApiListener * const, const boost::intrusive_ptr<icinga::Socket> &)) 0x156edc0 <icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr<icinga::Socket> const&)>, __t=@0x2a195f0: 0x7fffc0046f40, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:73
#4  0x00000000015982f7 in std::__invoke<void (icinga::ApiListener::*&)(boost::intrusive_ptr<icinga::Socket> const&), icinga::ApiListener*&, boost::intrusive_ptr<icinga::TcpSocket>&> (__fn=
---Type <return> to continue, or q <return> to quit---
    @0x2a195d8: (void (icinga::ApiListener::*)(icinga::ApiListener * const, const boost::intrusive_ptr<icinga::Socket> &)) 0x156edc0 <icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr<icinga::Socket> const&)>, __args=..., __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#5  0x00000000015982a8 in std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (this=0x2a195d8, __args=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#6  0x0000000001598236 in std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>::operator()<, void>() (
    this=0x2a195d8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#7  0x00000000015981fd in std::__invoke_impl<void, std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>>(std::__invoke_other, std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>&&) (__f=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#8  0x000000000159818d in std::__invoke<std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>>(std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>&&) (__fn=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#9  0x0000000001598165 in std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)> > >::_M_invoke<0ul> (this=0x2a195d8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:234
#10 0x0000000001598135 in std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)> > >::operator()() (this=0x2a195d8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:243
#11 0x0000000001597db9 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)> > > >::_M_run() (this=0x2a195d0) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:186
#12 0x00007ffff59ce57f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff69ee6db in start_thread (arg=0x7ffff7f83700) at pthread_create.c:463
#14 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 51 (Thread 0x7ffff7f42700 (LWP 18945)):
#0  0x00007ffff69f49f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fffc00473bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fffc0047368, cond=0x7fffc0047390) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fffc0047390, mutex=0x7fffc0047368) at pthread_cond_wait.c:655
#3  0x0000000001436df3 in boost::condition_variable::wait (this=0x7fffc0047368, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:81
#4  0x00000000014bdc08 in icinga::WorkQueue::WorkerThreadProc (this=0x7fffc0047310) at /home/michael/Coding/Icinga/icinga2/lib/base/workqueue.cpp:281
#5  0x00000000014c12e1 in std::__invoke_impl<void, void (icinga::WorkQueue::*&)(), icinga::WorkQueue*&> (
    __f=@0x29eeae8: (void (icinga::WorkQueue::*)(icinga::WorkQueue * const)) 0x14bda50 <icinga::WorkQueue::WorkerThreadProc()>, __t=@0x29eeaf8: 0x7fffc0047310)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:73
#6  0x00000000014c1212 in std::__invoke<void (icinga::WorkQueue::*&)(), icinga::WorkQueue*&> (
    __fn=@0x29eeae8: (void (icinga::WorkQueue::*)(icinga::WorkQueue * const)) 0x14bda50 <icinga::WorkQueue::WorkerThreadProc()>, __args=@0x29eeaf8: 0x7fffc0047310)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#7  0x00000000014c1a20 in std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x29eeae8, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#8  0x00000000014c19d6 in std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()>::operator()<, void>() (this=0x29eeae8)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#9  0x00000000014c190c in boost::detail::thread_data<std::_Bind<void (icinga::WorkQueue::*(icinga::WorkQueue*))()> >::run() (this=0x29ee930) at /usr/include/boost/thread/detail/thread.hpp:116
#10 0x00007ffff79bdbcd in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1
#11 0x00007ffff69ee6db in start_thread (arg=0x7ffff7f42700) at pthread_create.c:463
#12 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 34 (Thread 0x7ffff7f01700 (LWP 18918)):
#0  0x00007ffff69f4f85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ffff7f00ab8, expected=0, futex_word=0x24b501c <l_TimerCV+84>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7ffff7f00ab8, mutex=0x24b4fc8 <l_TimerCV>, cond=0x24b4ff0 <l_TimerCV+40>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x24b4ff0 <l_TimerCV+40>, mutex=0x24b4fc8 <l_TimerCV>, abstime=0x7ffff7f00ab8) at pthread_cond_wait.c:667
#3  0x000000000142bfe3 in boost::condition_variable::do_wait_until (this=0x24b4fc8 <l_TimerCV>, m=..., timeout=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:115
#4  0x00000000014287ac in boost::condition_variable::timed_wait (this=0x24b4fc8 <l_TimerCV>, m=..., abs_time=...) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:142
#5  0x000000000146ee16 in boost::condition_variable::timed_wait<boost::date_time::subsecond_duration<boost::posix_time::time_duration, 1000l> > (this=0x24b4fc8 <l_TimerCV>, m=..., wait_duration=...)
    at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:166
---Type <return> to continue, or q <return> to quit---
#6  0x000000000147550a in icinga::Timer::TimerThreadProc () at /home/michael/Coding/Icinga/icinga2/lib/base/timer.cpp:316
#7  0x0000000001477a57 in std::__invoke_impl<void, void (*)()> (__f=@0x29f18a8: 0x1475240 <icinga::Timer::TimerThreadProc()>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#8  0x00000000014779ed in std::__invoke<void (*)()> (__fn=@0x29f18a8: 0x1475240 <icinga::Timer::TimerThreadProc()>) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#9  0x00000000014779c5 in std::thread::_Invoker<std::tuple<void (*)()> >::_M_invoke<0ul> (this=0x29f18a8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:234
#10 0x0000000001477995 in std::thread::_Invoker<std::tuple<void (*)()> >::operator() (this=0x29f18a8) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:243
#11 0x0000000001477889 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run (this=0x29f18a0)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:186
#12 0x00007ffff59ce57f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff69ee6db in start_thread (arg=0x7ffff7f01700) at pthread_create.c:463
#14 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 33 (Thread 0x7ffff7ec0700 (LWP 18910)):
#0  0x00007ffff69f4f85 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7ffff7ebf568, expected=0, futex_word=0x24b2bf0 <icinga::Application::GetTP()::tp+480>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7ffff7ebf568, mutex=0x24b2ba0 <icinga::Application::GetTP()::tp+400>, cond=0x24b2bc8 <icinga::Application::GetTP()::tp+440>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x24b2bc8 <icinga::Application::GetTP()::tp+440>, mutex=0x24b2ba0 <icinga::Application::GetTP()::tp+400>, abstime=0x7ffff7ebf568) at pthread_cond_wait.c:667
#3  0x000000000142bfe3 in boost::condition_variable::do_wait_until (this=0x24b2ba0 <icinga::Application::GetTP()::tp+400>, m=..., timeout=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:115
#4  0x00000000014287ac in boost::condition_variable::timed_wait (this=0x24b2ba0 <icinga::Application::GetTP()::tp+400>, m=..., abs_time=...) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:142
#5  0x000000000146ee16 in boost::condition_variable::timed_wait<boost::date_time::subsecond_duration<boost::posix_time::time_duration, 1000l> > (this=0x24b2ba0 <icinga::Application::GetTP()::tp+400>, m=..., 
    wait_duration=...) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:166
#6  0x000000000146caf2 in icinga::ThreadPool::ManagerThreadProc (this=0x24b2a10 <icinga::Application::GetTP()::tp>) at /home/michael/Coding/Icinga/icinga2/lib/base/threadpool.cpp:239
#7  0x0000000001471be1 in std::__invoke_impl<void, void (icinga::ThreadPool::*&)(), icinga::ThreadPool*&> (
    __f=@0x2542338: (void (icinga::ThreadPool::*)(icinga::ThreadPool * const)) 0x146c950 <icinga::ThreadPool::ManagerThreadProc()>, __t=@0x2542348: 0x24b2a10 <icinga::Application::GetTP()::tp>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:73
#8  0x0000000001471b12 in std::__invoke<void (icinga::ThreadPool::*&)(), icinga::ThreadPool*&> (
    __fn=@0x2542338: (void (icinga::ThreadPool::*)(icinga::ThreadPool * const)) 0x146c950 <icinga::ThreadPool::ManagerThreadProc()>, __args=@0x2542348: 0x24b2a10 <icinga::Application::GetTP()::tp>)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#9  0x0000000001471ad0 in std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x2542338, __args=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:467
#10 0x0000000001471a86 in std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()>::operator()<, void>() (this=0x2542338)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/functional:549
#11 0x0000000001471a4d in std::__invoke_impl<void, std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()>>(std::__invoke_other, std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()>&&) (
    __f=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:60
#12 0x00000000014719dd in std::__invoke<std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()>>(std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()>&&) (__fn=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/bits/invoke.h:95
#13 0x00000000014719b5 in std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()> > >::_M_invoke<0ul> (this=0x2542338)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:234
#14 0x0000000001471985 in std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()> > >::operator()() (this=0x2542338)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:243
#15 0x0000000001471769 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ThreadPool::*(icinga::ThreadPool*))()> > > >::_M_run() (this=0x2542330)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/7.3.0/../../../../include/c++/7.3.0/thread:186
#16 0x00007ffff59ce57f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007ffff69ee6db in start_thread (arg=0x7ffff7ec0700) at pthread_create.c:463
#18 0x00007ffff508b88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7ffff7fcfd40 (LWP 18869)):
#0  0x00007ffff69f49f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fffe80024d8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fffe8002488, cond=0x7fffe80024b0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fffe80024b0, mutex=0x7fffe8002488) at pthread_cond_wait.c:655
#3  0x0000000001436df3 in boost::condition_variable::wait (this=0x7fffe8002488, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:81
---Type <return> to continue, or q <return> to quit---
#4  0x00000000014bd51e in icinga::WorkQueue::Join (this=0x7fffe8002380, stop=false) at /home/michael/Coding/Icinga/icinga2/lib/base/workqueue.cpp:130

#5  0x0000000001b924f2 in icinga::GraphiteWriter::Pause (this=0x7fffe80021e0) at /home/michael/Coding/Icinga/icinga2/lib/perfdata/graphitewriter.cpp:110
#6  0x00000000013131ce in icinga::ConfigObject::SetAuthority (this=0x7fffe80021e0, authority=false) at /home/michael/Coding/Icinga/icinga2/lib/base/configobject.cpp:478
#7  0x00000000013135a6 in icinga::ConfigObject::Deactivate (this=0x7fffe80021e0, runtimeRemoved=false) at /home/michael/Coding/Icinga/icinga2/lib/base/configobject.cpp:416
#8  0x00000000013156e4 in icinga::ConfigObject::StopObjects () at /home/michael/Coding/Icinga/icinga2/lib/base/configobject.cpp:627
#9  0x00000000012ce5cc in icinga::Application::SigUsr2Handler () at /home/michael/Coding/Icinga/icinga2/lib/base/application.cpp:755
#10 <signal handler called>
#11 0x00007ffff504e9d0 in __GI___nanosleep (requested_time=requested_time@entry=0x7fffffff7c50, remaining=remaining@entry=0x7fffffff7c50) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#12 0x00007ffff504e8aa in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55
#13 0x0000000001492b96 in icinga::Utility::Sleep (timeout=2.5) at /home/michael/Coding/Icinga/icinga2/lib/base/utility.cpp:428
#14 0x00000000012cb620 in icinga::Application::RunEventLoop (this=0x7fffe0007b10) at /home/michael/Coding/Icinga/icinga2/lib/base/application.cpp:315
#15 0x0000000001864c2e in icinga::IcingaApplication::Main (this=0x7fffe0007b10) at /home/michael/Coding/Icinga/icinga2/lib/icinga/icingaapplication.cpp:119
#16 0x00000000012cffe6 in icinga::Application::Run (this=0x7fffe0007b10) at /home/michael/Coding/Icinga/icinga2/lib/base/application.cpp:1033
#17 0x000000000167b092 in icinga::DaemonCommand::Run (this=0x252e840, vm=..., ap=warning: Type size unknown, assuming 1. Try casting to a known type, or void *.

warning: Type size unknown, assuming 1. Try casting to a known type, or void *.
std::vector of length 0, capacity 0) at /home/michael/Coding/Icinga/icinga2/lib/cli/daemoncommand.cpp:320
#18 0x00000000012b53f6 in Main () at /home/michael/Coding/Icinga/icinga2/icinga-app/icinga.cpp:681
#19 0x00000000012afa2c in main (argc=3, argv=0x7fffffffe3b8) at /home/michael/Coding/Icinga/icinga2/icinga-app/icinga.cpp:975

You can see thread 1 is handling the SigUsr2 signal which is used to hand over to the new process. Currently it tries to stop the GraphiteWriter and waits for it to finally finish.

Having a look into thread 55, which is the thread which handles the GraphiteWriter. Thread 55 currently waits for a lock.

#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007ffff69f1098 in __GI___pthread_mutex_lock (mutex=0x2c3ab20) at ../nptl/pthread_mutex_lock.c:113
#2  0x00000000013e7455 in boost::recursive_mutex::lock (this=0x2c3ab20) at /usr/include/boost/thread/pthread/recursive_mutex.hpp:113
#3  0x00000000013e736b in icinga::ObjectLock::LockMutex (object=0x7fffe80021e0) at /home/michael/Coding/Icinga/icinga2/lib/base/objectlock.cpp:62
#4  0x00000000013e724e in icinga::ObjectLock::Lock (this=0x7ffff1e1aab0) at /home/michael/Coding/Icinga/icinga2/lib/base/objectlock.cpp:88
#5  0x00000000013e72d6 in icinga::ObjectLock::ObjectLock (this=0x7ffff1e1aab0, object=0x7fffe80021e0) at /home/michael/Coding/Icinga/icinga2/lib/base/objectlock.cpp:44
#6  0x0000000001b9458a in icinga::GraphiteWriter::SendMetric (this=0x7fffe80021e0, prefix=..., name=..., value=73.799999999999997, ts=1541953843.286433)

Finding out which thread holds the lock on which thread 55 waits.

(gdb) f 3
#3  0x00000000013e736b in icinga::ObjectLock::LockMutex (object=0x7fffe80021e0) at /home/michael/Coding/Icinga/icinga2/lib/base/objectlock.cpp:62
62              mtx->lock();
(gdb) p mtx
$3 = (boost::recursive_mutex *) 0x2c3ab20
(gdb) p *(boost::recursive_mutex *) 0x2c3ab20
$4 = {m = pthread_mutex_t = {Type = Recursive, Status = Acquired, possibly with waiters, Owner ID = 18869, Robust = No, Shared = No, Protocol = None, Times acquired by the owner = 2}}
(gdb) thread find 18869
Thread 1 has target id 'Thread 0x7ffff7fcfd40 (LWP 18869)'
(gdb)

Thread 55 waits for a lock that is hold by thread 1.

So we have a deadlock.

Possible solution

Do not lock the GraphiteWriter object in SendMetric().

diff --git a/lib/perfdata/graphitewriter.cpp b/lib/perfdata/graphitewriter.cpp
index a01f6c48b..1b6461d51 100644
--- a/lib/perfdata/graphitewriter.cpp
+++ b/lib/perfdata/graphitewriter.cpp
@@ -299,7 +299,7 @@ void GraphiteWriter::SendMetric(const String& prefix, const String& name, double
        msgbuf << "\n";
        String metric = msgbuf.str();

-       ObjectLock olock(this);
+       boost::mutex::scoped_lock lock(m_StreamMutex);

        if (!GetConnected())
                return;
diff --git a/lib/perfdata/graphitewriter.hpp b/lib/perfdata/graphitewriter.hpp
index 6934d13b3..ec4c5bbfe 100644
--- a/lib/perfdata/graphitewriter.hpp
+++ b/lib/perfdata/graphitewriter.hpp
@@ -27,6 +27,7 @@
 #include "base/timer.hpp"
 #include "base/workqueue.hpp"
 #include <fstream>
+#include <boost/thread/mutex.hpp>

 namespace icinga
 {
@@ -54,6 +55,7 @@ protected:

 private:
        Stream::Ptr m_Stream;
+       boost::mutex m_StreamMutex;
        WorkQueue m_WorkQueue{10000000, 1};

        Timer::Ptr m_ReconnectTimer;

This change removes the lock from the GraphiteWriter but it should lock the stream which actually send the metrics.

Before putting this in a PR I'll leave this here for further feedback and to get a clear mind about the problem.

mcktr commented 6 years ago

Further analysis.

The bug was introduced/got visible by af6c8a1f50aa06c07d58e2cb00cf93f010405f6b. Reverting the change resolves the bug immediately.

git revert af6c8a1f50aa06c07d58e2cb00cf93f010405f6b

[2018-11-12 19:45:46 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 134, rate: 348.45/s (20907/min 20907/5min 20907/15min);
[2018-11-12 19:45:46 +0100] information/WorkQueue: #6 (IdoMysqlConnection, ido-mysql) items: 60618, rate: 277.3/s (16638/min 16638/5min 16638/15min); empty in 17847 days, 18 hours, 45 minutes and 46 seconds
[2018-11-12 19:45:46 +0100] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 1230.27/s (73816/min 73816/5min 73816/15min);
[2018-11-12 19:45:46 +0100] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-12 19:45:51 +0100] information/Application: Got reload command: Starting new instance.
[2018-11-12 19:45:53 +0100] information/Application: Reload requested, letting new process take over.
[2018-11-12 19:45:54 +0100] information/FileLogger: 'main-log' started.
[2018-11-12 19:45:54 +0100] information/FileLogger: 'debug-file' started.
[2018-11-12 19:45:54 +0100] information/ApiListener: 'api' started.
[2018-11-12 19:45:54 +0100] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2018-11-12 19:45:54 +0100] information/ExternalCommandListener: 'command' started.
[2018-11-12 19:45:54 +0100] information/CheckerComponent: 'checker' started.
[2018-11-12 19:45:54 +0100] information/DbConnection: 'ido-mysql' started.
[2018-11-12 19:45:54 +0100] information/ConfigItem: Activated all objects.
[2018-11-12 19:45:54 +0100] information/GraphiteWriter: 'graphite' resumed.
[2018-11-12 19:45:54 +0100] information/GraphiteWriter: Finished reconnecting to Graphite in 0.000531912 second(s).
[2018-11-12 19:45:54 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2018-11-12 19:45:54 +0100] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2018-11-12 19:45:54 +0100] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2018-11-12 19:46:02 +0100] information/WorkQueue: #5 (GraphiteWriter, graphite) items: 381, rate: 286.767/s (17206/min 17206/5min 17206/15min);
[2018-11-12 19:46:02 +0100] information/WorkQueue: #6 (IdoMysqlConnection, ido-mysql) items: 52214, rate: 208.35/s (12501/min 12501/5min 12501/15min); empty in 17847 days, 18 hours, 46 minutes and 2 seconds
[2018-11-12 19:46:02 +0100] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-12 19:46:02 +0100] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 1047.7/s (62862/min 62862/5min 62862/15min);

I am pretty sure this not only happens in setups with much hosts/services, but also in setup with only a few hosts/services. It is more noticeable in a setup with many hosts/services since the GraphiteWriter writes more performance data, whereby the GraphiteWriter thread is more often busy with SendMetric() which leads to the described deadlock.

Reverting af6c8a1f50aa06c07d58e2cb00cf93f010405f6b is not really an option here. I'll prepare a PR for the proposed fix.

brudrafon commented 6 years ago

Thank you all guys for quick debug and fix, works like a charm. :)

Icinga / icinga2

Icinga2.service state is reloading in systemd after safe-reload until systemd time-out #6753

Logs - Systemd

Logs - Icinga2

Service status in systemd

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

Analysis

Possible solution