search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2k stars 574 forks source link

Logrotate issues since 2.10.1-1 #6737

Closed sschindel closed 5 years ago

sschindel commented 5 years ago

Hey!

Since upgrading to 2.10.1-1 we face some issues with the logrotating part. The daemon sometime crashes after the logrotate run, this did not happen before. This is the crash report:

cat /var/log/icinga2/crash/report.1540790715.016844

Application version: r2.10.1-1

System information: Platform: Debian GNU/Linux Platform version: 9 (stretch) Kernel: Linux Kernel version: 4.9.0-6-amd64 Architecture: x86_64

Build information: Compiler: GNU 6.3.0 Build host: 1880c6977367

Application information:

General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2

Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var

Internal paths: Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid Stacktrace:

(0) libc.so.6: gsignal (+0xcf) [0x7fa1362c5fff]
(1) libc.so.6: abort (+0x16a) [0x7fa1362c742a]
(2) libc.so.6: <unknown function> (+0x70c00) [0x7fa136303c00]
(3) libc.so.6: <unknown function> (+0x76fc6) [0x7fa136309fc6]
(4) libc.so.6: <unknown function> (+0x7780e) [0x7fa13630a80e]
(5) icinga2: std::_Function_base::_Base_manager<std::_Bind<std::_Mem_fn<void (icinga::Timer::*)()> (boost::intrusive_ptr<icinga::Timer>)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (+0x38) [0x563e339d0358]
(6) icinga2: icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (+0x4ca) [0x563e339c116a]
(7) libboost_thread.so.1.62.0: <unknown function> (+0x12116) [0x7fa138c1f116]
(8) libpthread.so.0: <unknown function> (+0x7494) [0x7fa137c41494]
(9) libc.so.6: clone (+0x3f) [0x7fa13637bacf]

Failed to launch GDB: No such file or directory

Crunsher commented 5 years ago

A description of the circumstances to reproduce this is required. You say it happens during log rotate, what makes you think that?

The log around this time would also be very welcome.

sschindel commented 5 years ago

We've got a stock Debian cron and logrotate setup, around 200 Hosts and randomly one or two of those hosts has this issue. Logrotate in in cron.daily, which gets executed at 06:25:

25 6 * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

/var/log/icinga2# ls -l total 1624 drwxr-s--x 3 nagios adm 4096 Mar 7 2017 compat drwxr-s--x 2 nagios adm 4096 Oct 29 06:25 crash -rw------- 1 nagios adm 0 Mar 7 2017 icinga2.err -rw-r--r-- 1 nagios nagios 57755 Oct 29 10:35 icinga2.log -rw-r--r-- 1 nagios nagios 1301697 Oct 29 06:25 icinga2.log.1 -rw-r--r-- 1 nagios nagios 48305 Oct 28 06:25 icinga2.log.2.gz -rw-r--r-- 1 nagios nagios 42199 Oct 27 06:25 icinga2.log.3.gz -rw-r--r-- 1 nagios nagios 42196 Oct 26 06:25 icinga2.log.4.gz -rw-r--r-- 1 nagios nagios 42458 Oct 25 06:25 icinga2.log.5.gz -rw-r--r-- 1 nagios nagios 41915 Oct 24 06:25 icinga2.log.6.gz -rw-r--r-- 1 nagios nagios 43024 Oct 23 06:25 icinga2.log.7.gz -rw-r--r-- 1 root adm 6361 Mar 7 2017 startup.log

The crash report is from that time:

stat /var/log/icinga2/crash/report.1540790715.016844

File: /var/log/icinga2/crash/report.1540790715.016844 Size: 2148 Blocks: 8 IO Block: 4096 regular file Device: 802h/2050d Inode: 1711371 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 82/ nagios) Gid: ( 4/ adm) Access: 2018-10-29 10:03:38.490438878 +0100 Modify: 2018-10-29 06:25:15.105257350 +0100 Change: 2018-10-29 06:25:15.105257350 +0100 Birth: -

And the last entry of the log files are:

[2018-10-29 06:23:17 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:23:25 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (30/min 150/5min 450/15min); [2018-10-29 06:23:26 +0100] information/WorkQueue: #15 (JsonRpcConnection, #6) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:23:35 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (18/min 90/5min 270/15min); [2018-10-29 06:23:36 +0100] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:23:37 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:23:45 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (12/min 60/5min 180/15min); [2018-10-29 06:23:45 +0100] information/WorkQueue: #13 (JsonRpcConnection, #4) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:23:55 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:23:57 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:24:05 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (30/min 150/5min 450/15min); [2018-10-29 06:24:06 +0100] information/WorkQueue: #11 (JsonRpcConnection, #2) items: 0, rate: 1.33333/s (80/min 400/5min 1200/15min); [2018-10-29 06:24:07 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:24:15 +0100] information/WorkQueue: #16 (JsonRpcConnection, #7) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:24:15 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (36/min 180/5min 540/15min); [2018-10-29 06:24:25 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (30/min 150/5min 450/15min); [2018-10-29 06:24:35 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (18/min 90/5min 270/15min); [2018-10-29 06:24:45 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (12/min 60/5min 180/15min); [2018-10-29 06:24:55 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:25:05 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (30/min 150/5min 450/15min);

Do you need any further information?

fladna9 commented 5 years ago

I think i got the same problem on a machine i host.

root@log:/home/user# cat /var/log/icinga2/crash/report.1540790722.611491
  Application version: r2.10.1-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-7-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 1880c6977367

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7f3a80164fff]
    (1) libc.so.6: abort (+0x16a) [0x7f3a8016642a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7f3a801a2c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7f3a801a8fc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7f3a801a980e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa6) [0x55f96871f5c6]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x55f9686d5f0a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x55f9686b33a2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x55f9688e7974]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x55f9686b1351]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x55f968860259]
    (11) icinga2: <unknown function> (+0x266ef6) [0x55f9686b6ef6]
    (12) icinga2: main (+0x49) [0x55f96865b299]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7f3a801522e1]
    (14) icinga2: _start (+0x2a) [0x55f96867881a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory
root@log:/home/user# stat /var/log/icinga2/crash/report.1540790722.611491
  File: /var/log/icinga2/crash/report.1540790722.611491
  Size: 2517        Blocks: 8          IO Block: 4096   regular file
Device: 801h/2049d  Inode: 12323212    Links: 1
Access: (0644/-rw-r--r--)  Uid: (  112/  nagios)   Gid: (    4/     adm)
Access: 2018-10-30 07:51:47.899211999 +0100
Modify: 2018-10-29 06:25:23.771421824 +0100
Change: 2018-10-29 06:25:23.771421824 +0100
 Birth: -

Last log lines:

[2018-10-29 06:20:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:21:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:21:22 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:21:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:22:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:22:22 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:22:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:23:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:23:22 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:23:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:24:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:24:22 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:24:22 +0100] information/WorkQueue: #10 (JsonRpcConnection, #1) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-10-29 06:24:23 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-10-29 06:24:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:24:52 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-10-29 06:24:52 +0100] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0.3/s (18/min 90/5min 270/15min);
[2018-10-29 06:24:52 +0100] information/WorkQueue: #13 (JsonRpcConnection, #4) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-10-29 06:24:53 +0100] information/WorkQueue: #15 (JsonRpcConnection, #6) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-10-29 06:25:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:25:02 +0100] information/WorkQueue: #11 (JsonRpcConnection, #2) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-10-29 06:25:12 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-10-29 06:25:19 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2018-10-29 06:25:23 +0100] information/WorkQueue: #16 (JsonRpcConnection, #7) items: 0, rate:  0/s (0/min 0/5min 0/15min);

I happens randomly on other machine, just like @sschindel, but the stacktrace is not exactly the same... Can provide more info if it can help.

Oh, and about configuration, nothing has been manually changed. Only launched a # icinga2 node wizard to connect to central Icinga2 monitoring system.

Thank you very much.

Edit: High probability this is a double of #6736

dnsmichi commented 5 years ago

Are you using SysVinit or Systemd here? I'm thinking about a relationship to #6356.

dnsmichi commented 5 years ago

If it is just Debian 9, #6571 could come into play as well.

fladna9 commented 5 years ago

@dnsmichi : Standard Debian Stretch install, so systemd :-) I also have 3 or 4 Ubuntu VMs using systemd as well (compared to about 30+ Debian Stretch machines), but no crash so far.

Hope it helps.

sschindel commented 5 years ago

Hi,

we are also using Debian Stretch and had no issues with 2.9.2-1. This issue only appeared since upgrading to 2.10.1-1. We are using the packages from packges.icinga.com:

deb http://packages.icinga.com/debian icinga-stretch main

fladna9 commented 5 years ago

I'm still using stable deb source.list, so still using the sometimes-crashing-on-my-vm version.

It did not happen for a week, but then it happened today again when logrotate was called again. My Icinga2 (slave) node is out again. Strange thing: it is always the same machine.

We host about 40 VMs and hosts (35 Debian Stretch (Systemd), 1 Debian Jessie (Systemd), rest is Ubuntu (Systemd)), and only 1 is posing problems.

Exactly the same time as previously (6:25 AM CET, that is to say 5:25 AM UTC, some machines aren't on the same timezone, but i've taken care to take the correct log lines).

Here is my syslog: image

Here is crash report:

root# cat /var/log/icinga2/crash/report.1541481925.521074
  Application version: r2.10.1-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-7-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 1880c6977367

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7fc551ebdfff]
    (1) libc.so.6: abort (+0x16a) [0x7fc551ebf42a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7fc551efbc00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7fc551f01fc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7fc551f0280e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa6) [0x562c3d7c25c6]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x562c3d778f0a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x562c3d7563a2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x562c3d98a974]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x562c3d754351]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x562c3d903259]
    (11) icinga2: <unknown function> (+0x266ef6) [0x562c3d759ef6]
    (12) icinga2: main (+0x49) [0x562c3d6fe299]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7fc551eab2e1]
    (14) icinga2: _start (+0x2a) [0x562c3d71b81a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

Here are the last /var/log/icinga2/icinga2.log.1file corresponding to before the crash.

[2018-11-06 06:20:18 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:20:36 +0100] information/WorkQueue: #16 (JsonRpcConnection, #7) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:20:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:20:56 +0100] information/WorkQueue: #11 (JsonRpcConnection, #2) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:20:56 +0100] information/WorkQueue: #15 (JsonRpcConnection, #6) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:20:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:21:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:21:36 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:21:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:21:56 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:21:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:22:06 +0100] information/WorkQueue: #10 (JsonRpcConnection, #1) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:22:16 +0100] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0.3/s (18/min 90/5min 270/15min);
[2018-11-06 06:22:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:22:22 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2018-11-06 06:22:28 +0100] information/WorkQueue: #13 (JsonRpcConnection, #4) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:22:38 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-06 06:22:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:22:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:23:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:23:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:23:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:24:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:24:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:24:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-06 06:25:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);

Here is the systemctl status icinga2 result

● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/icinga2.service.d
           └─limits.conf
   Active: failed (Result: signal) since Tue 2018-11-06 06:25:25 CET; 2h 0min ago
  Process: 9187 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=killed, signal=ABRT)
  Process: 9177 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 9187 (code=killed, signal=ABRT)

Nov 06 06:25:25 log systemd[1]: icinga2.service: Main process exited, code=killed, status=6/ABRT
Nov 06 06:25:25 log systemd[1]: icinga2.service: Unit entered failed state.
Nov 06 06:25:25 log systemd[1]: icinga2.service: Failed with result 'signal'.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Here is the /etc/logrotate.d/icinga2 file.

root# cat /etc/logrotate.d/icinga2
/var/log/icinga2/icinga2.log /var/log/icinga2/debug.log {
    daily
    rotate 7
    su nagios nagios
    compress
    delaycompress
    missingok
    notifempty
    create 644 nagios nagios
    postrotate
        /bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
    endscript
}

/var/log/icinga2/error.log {
    daily
    rotate 90
    su nagios nagios
    compress
    delaycompress
    missingok
    notifempty
    create 644 nagios nagios
    # TODO: figure out how to get Icinga to re-open this log file
}

Hope it helps.

To dev staff: Here to help/test things if needed ;-)

Edit: added systemctl status icinga2 output

dnsmichi commented 5 years ago

This looks like a wrong file descriptor handle, i.e. logrotate moves something away where Icinga still wants to write to. I suspect the close-stdio changes and flushing the streams to cause trouble here.

If anyone has time to lookup the code changes, I‘m out for OSMC this week.

NielsH commented 5 years ago

Hi,

I also have this issue, please see my crash log as well. This also includes the gdb output:

 Application version: r2.10.1-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-8-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 1880c6977367

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7fbf8b938fff]
    (1) libc.so.6: abort (+0x16a) [0x7fbf8b93a42a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7fbf8b976c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7fbf8b97cfc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7fbf8b97d80e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa6) [0x560aed2475c6]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x560aed1fdf0a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x560aed1db3a2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x560aed40f974]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x560aed1d9351]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x560aed388259]
    (11) icinga2: <unknown function> (+0x266ef6) [0x560aed1deef6]
    (12) icinga2: main (+0x49) [0x560aed183299]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7fbf8b9262e1]
    (14) icinga2: _start (+0x2a) [0x560aed1a081a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

[New LWP 31300]
[New LWP 31301]
[New LWP 31302]
[New LWP 31303]
[New LWP 31304]
[New LWP 31316]
[New LWP 31326]
[New LWP 31327]
[New LWP 31353]
[New LWP 31354]
[New LWP 31355]
[New LWP 31356]
[New LWP 31357]
[New LWP 31358]
[New LWP 31359]
[New LWP 31360]
[New LWP 31361]
[New LWP 31362]
[New LWP 31363]
[New LWP 31364]
[New LWP 31365]
[New LWP 31366]
[New LWP 31367]
[New LWP 31368]
[New LWP 31420]
[New LWP 31485]
[New LWP 31486]
[New LWP 31487]
[New LWP 31488]
[New LWP 6596]
[New LWP 20511]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fbf8d2bdb3a in __waitpid (pid=15301, stat_loc=0x7ffca6d8b7bc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29  ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.

Thread 32 (Thread 0x7fbf73910700 (LWP 20511)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73910700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73910700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460254365440, 4463560644109586558, 0, 140460254893855, 140460254365440, 140460061493968, -4427276064668987266, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 31 (Thread 0x7fbf7388e700 (LWP 6596)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7388e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7388e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460253832960, 4463560644109586558, 0, 140460254893855, 140460253832960, 140460061493184, -4427276132314722178, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 30 (Thread 0x7fbf73339700 (LWP 31488)):
#0  0x00007fbf8b9e567d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed2111ae in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73339700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73339700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460248241920, 4463560644109586558, 0, 140460253818223, 0, 140460707086400, -4427277410604363650, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 29 (Thread 0x7fbf7337a700 (LWP 31487)):
#0  0x00007fbf8b9e567d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed2111ae in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7337a700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7337a700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460248508160, 4463560644109586558, 0, 140460253818223, 0, 140460707086400, -4427277375707754370, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 28 (Thread 0x7fbf733bb700 (LWP 31486)):
#0  0x00007fbf8b9e567d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed2111ae in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf733bb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf733bb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460248774400, 4463560644109586558, 0, 140460253818223, 0, 140460707086400, -4427277342958628738, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 27 (Thread 0x7fbf733fc700 (LWP 31485)):
#0  0x00007fbf8b9e567d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed2111ae in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf733fc700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf733fc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460249040640, 4463560644109586558, 0, 140460253818223, 0, 140460707086400, -4427277308062019458, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 26 (Thread 0x7fbf7343d700 (LWP 31420)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed3179c8 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x0000560aed31877b in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x0000560aed26213e in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x0000560aed26ac85 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7343d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7343d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460249306880, 4463560644109586558, 0, 140460253820031, 0, 140460707086400, -4427277825068707714, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 25 (Thread 0x7fbf7347e700 (LWP 31368)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed26aebb in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7347e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7347e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460249573120, 4463560644109586558, 0, 140460253820927, 0, 140460707086400, -4427277790172098434, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 24 (Thread 0x7fbf734bf700 (LWP 31367)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed26aebb in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf734bf700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf734bf700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460249839360, 4463560644109586558, 0, 140460253820927, 0, 140460707086400, -4427277757422972802, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 23 (Thread 0x7fbf73500700 (LWP 31366)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed26aebb in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73500700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73500700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460250105600, 4463560644109586558, 0, 140460253820927, 0, 140460707086400, -4427277722526363522, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 22 (Thread 0x7fbf73541700 (LWP 31365)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed26aebb in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73541700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73541700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460250371840, 4463560644109586558, 0, 140460253820927, 0, 140460707086400, -4427277672597368706, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 21 (Thread 0x7fbf73582700 (LWP 31364)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed3179c8 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x0000560aed31877b in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x0000560aed26213e in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x0000560aed26ac85 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73582700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73582700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460250638080, 4463560644109586558, 0, 140460253820031, 0, 140460707086400, -4427277637700759426, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 20 (Thread 0x7fbf735c3700 (LWP 31363)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf735c3700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf735c3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460250904320, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427277604951633794, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 19 (Thread 0x7fbf73604700 (LWP 31362)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73604700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73604700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460251170560, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427278119810838402, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 18 (Thread 0x7fbf73645700 (LWP 31361)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73645700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73645700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460251436800, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427278087061712770, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 17 (Thread 0x7fbf73686700 (LWP 31360)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73686700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73686700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460251703040, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427278052165103490, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 16 (Thread 0x7fbf736c7700 (LWP 31359)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf736c7700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf736c7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460251969280, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427278019415977858, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 15 (Thread 0x7fbf73708700 (LWP 31358)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73708700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73708700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460252235520, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427277984519368578, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 14 (Thread 0x7fbf73749700 (LWP 31357)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73749700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73749700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460252501760, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427277951770242946, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 13 (Thread 0x7fbf7378a700 (LWP 31356)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7378a700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7378a700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460252768000, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427277916873633666, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 12 (Thread 0x7fbf737cb700 (LWP 31355)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf737cb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf737cb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460253034240, 4463560644109586558, 0, 140460254893855, 0, 140460707086400, -4427277884124508034, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 11 (Thread 0x7fbf7380c700 (LWP 31354)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed241506 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7380c700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7380c700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460253300480, 4463560644109586558, 0, 140460254894383, 0, 140460707086400, -4427276199960457090, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 10 (Thread 0x7fbf7384d700 (LWP 31353)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7384d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7384d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460253566720, 4463560644109586558, 0, 140460254893743, 0, 140460707086400, -4427276167211331458, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 9 (Thread 0x7fbf73951700 (LWP 31327)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed498f7b in icinga::CheckerComponent::CheckThreadProc() ()
No symbol table info available.
#3  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73951700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73951700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460254631680, 4463560644109586558, 0, 140723107711823, 140460254631680, 94604956656800, -4427276014739992450, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 8 (Thread 0x7fbf73992700 (LWP 31326)):
#0  0x00007fbf8d2bd3ed in accept () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23af57 in icinga::Socket::Accept() ()
No symbol table info available.
#2  0x0000560aed2b9a97 in icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr<icinga::Socket> const&) ()
No symbol table info available.
#3  0x0000560aed370d06 in ?? ()
No symbol table info available.
#4  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#5  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73992700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73992700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460254897920, 4463560644109586558, 0, 140723107709919, 140460254897920, 94604956605632, -4427275979843383170, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 7 (Thread 0x7fbf739d3700 (LWP 31316)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
No locals.
#1  0x0000560aed242833 in icinga::Timer::TimerThreadProc() ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf739d3700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf739d3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460255164160, 4463560644109586558, 0, 140723107712367, 0, 140460707086400, -4427275947094257538, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 6 (Thread 0x7fbf8e71e700 (LWP 31304)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed241506 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e71e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e71e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460705310464, 4463560644109586558, 0, 140723107732495, 0, 140460707086400, -4427359343011114882, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 5 (Thread 0x7fbf8e75f700 (LWP 31303)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed2ad41d in icinga::ApiListener::ApiTimerHandler() ()
No symbol table info available.
#5  0x0000560aed25e931 in ?? ()
No symbol table info available.
#6  0x0000560aed220a14 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x0000560aed2440d8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e75f700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e75f700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460705576704, 4463560644109586558, 0, 140723107731855, 0, 140460707086400, -4427359310261989250, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 4 (Thread 0x7fbf8e822700 (LWP 31302)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed49773e in icinga::CheckerComponent::ResultTimerHandler() ()
No symbol table info available.
#5  0x0000560aed25e931 in ?? ()
No symbol table info available.
#6  0x0000560aed220a14 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x0000560aed2440d8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e822700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e822700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460706375424, 4463560644109586558, 0, 140723107731855, 140460706375424, 94604950740848, -4427357541272334210, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 3 (Thread 0x7fbf8e7e1700 (LWP 31301)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e7e1700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e7e1700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460706109184, 4463560644109586558, 0, 140723107731855, 140460706109184, 94604950813008, -4427359225436385154, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 2 (Thread 0x7fbf8e7a0700 (LWP 31300)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e7a0700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e7a0700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460705842944, 4463560644109586558, 0, 140723107731855, 140460705842944, 94604950813920, -4427359275365379970, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 1 (Thread 0x7fbf8e82e480 (LWP 31268)):
#0  0x00007fbf8d2bdb3a in __waitpid (pid=15301, stat_loc=0x7ffca6d8b7bc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
        resultvar = 18446744073709551104
        sc_cancel_oldtype = 0
#1  0x0000560aed1b528e in icinga::Application::AttachDebugger(icinga::String const&, bool) ()
No symbol table info available.
#2  0x0000560aed1d99a7 in icinga::Application::SigAbrtHandler(int) ()
No symbol table info available.
#3  <signal handler called>
No locals.
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {0, 2314885530818453536, 8660248813382869024, 7596496373740942904, 3419760881481315694, 7310595060593224044, 3328211180835464301, 7090415180238960433, 3472328515408705638, 7293692169334238512, 8223625903103632184, 3472328295963438455, 4192904167887482928, 7365366848950382640, 3472665881546876514, 4064043904900345904}}
        pid = <optimized out>
        tid = <optimized out>
#5  0x00007fbf8b93a42a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x756e672d78756e69, sa_sigaction = 0x756e672d78756e69}, sa_mask = {__val = {3761119431852583983, 7377571526170604334, 3472328301395600696, 4063716251160426285, 8606977229197423204, 3472328296226648109, 3475143045726351408, 7378645556122361904, 3472331625700155747, 3918522442241748272, 8223625903103818084, 3472328295963438455, 4192904167887482928, 2314885531086893104, 2314885530818453536, 140723107711424}}, sa_flags = 123, sa_restorer = 0x7ffca6d8c9c0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#6  0x00007fbf8b976c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fbf8ba6bd98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
        ap = {{gp_offset = 40, fp_offset = 32703, overflow_arg_area = 0x7ffca6d8c9d0, reg_save_area = 0x7ffca6d8c960}}
        fd = 2
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#7  0x00007fbf8b97cfc6 in malloc_printerr (action=3, str=0x7fbf8ba6bef0 "double free or corruption (fasttop)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
        buf = "0000560aeed709c0"
        cp = <optimized out>
        ar_ptr = <optimized out>
        ptr = <optimized out>
        str = 0x7fbf8ba6bef0 "double free or corruption (fasttop)"
        action = 3
#8  0x00007fbf8b97d80e in _int_free (av=0x7fbf8bc9fb00 <main_arena>, p=0x560aeed709b0, have_lock=0) at malloc.c:3905
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = <optimized out>
        locked = <optimized out>
        __func__ = "_int_free"
#9  0x0000560aed2475c6 in icinga::StreamLogger::BindStream(std::ostream*, bool) ()
No symbol table info available.
#10 0x0000560aed1fdf0a in icinga::FileLogger::ReopenLogFile() ()
No symbol table info available.
#11 0x0000560aed1db3a2 in icinga::Application::RunEventLoop() ()
No symbol table info available.
#12 0x0000560aed40f974 in icinga::IcingaApplication::Main() ()
No symbol table info available.
#13 0x0000560aed1d9351 in icinga::Application::Run() ()
No symbol table info available.
#14 0x0000560aed388259 in icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const ()
No symbol table info available.
#15 0x0000560aed1deef6 in ?? ()
No symbol table info available.
#16 0x0000560aed183299 in main ()
No symbol table info available.

After the crash a new icinga2.log is already created:

root@icingasql02:~# ls -lah /var/log/icinga2/
total 1.1M
drwxr-s--x  4 nagios adm    4.0K Nov  7 06:25 .
drwxr-xr-x 14 root   root   4.0K Nov  7 06:25 ..
drwxr-s--x  3 nagios adm    4.0K Oct 17  2017 compat
drwxr-s--x  2 nagios adm    4.0K Nov  7 06:25 crash
-rw-r--r--  1 nagios nagios    0 Nov  7 06:25 icinga2.log
-rw-r--r--  1 nagios nagios 838K Nov  7 06:25 icinga2.log.1
-rw-r--r--  1 nagios nagios  29K Nov  6 06:25 icinga2.log.2.gz
-rw-r--r--  1 nagios nagios  26K Nov  5 06:25 icinga2.log.3.gz
-rw-r--r--  1 nagios nagios  26K Nov  4 06:25 icinga2.log.4.gz
-rw-r--r--  1 nagios nagios  26K Nov  3 06:24 icinga2.log.5.gz
-rw-r--r--  1 nagios nagios  30K Nov  2 06:24 icinga2.log.6.gz
-rw-r--r--  1 nagios nagios  30K Nov  1 06:24 icinga2.log.7.gz

But as you can see nothing is written to it yet. When I start up the Icinga2 daemon again after the crash it starts writing to icinga2.log

mcktr commented 5 years ago

Can you try the snapshot packages? In #6736 a bug was fixed which seems similar to this.

fladna9 commented 5 years ago

Hello @mcktr, Is snapshot deb repository stable enough to go on a production server? Because my crashing machine is an internal-but-important-on-production server...

dnsmichi commented 5 years ago

@mcktr the changes you are referring to are solely cluster related, nothing which points to the stream logger class. I've fixed a possible double free as see in the gdb stacktrace yesterday, please test them in the snapshot packages once being built. On a related note, right now the master solely contains bugfixes and no radical changes, so it is safe to assume it works.

dnsmichi commented 5 years ago

Thus far, I am not able to reproduce this with 2.10.1 on Debian Stretch.

wget -O - https://packages.icinga.com/icinga.key | apt-key add -

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
 echo "deb http://packages.icinga.com/debian icinga-${DIST} main" > \
 /etc/apt/sources.list.d/${DIST}-icinga.list
 echo "deb-src http://packages.icinga.com/debian icinga-${DIST} main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga.list

apt-get update

apt-get install -y icinga2 vim

vim /etc/logrotate.d/icinga2

/var/log/icinga2/icinga2.log /var/log/icinga2/debug.log {
    size 1K
    rotate 7
    su nagios nagios
    compress
    delaycompress
    missingok
    notifempty
    create 644 nagios nagios
    postrotate
        /bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
    endscript
}

/var/log/icinga2/error.log {
    size 1K
    rotate 90
    su nagios nagios
    compress
    delaycompress
    missingok
    notifempty
    create 644 nagios nagios
    # TODO: figure out how to get Icinga to re-open this log file
}

logrotate -d /etc/logrotate.d/icinga2
sschindel commented 5 years ago

Has this already been released? There was an update some days ago, but this one still does not fix that issue.

fladna9 commented 5 years ago

@sschindel Same, crashed this morning as well.

dnsmichi commented 5 years ago

Please add more logs from such crashes and everything else which helps reproduce the issue. I wasn't able to reproduce this yet.

fladna9 commented 5 years ago

Today's logs

File /var/log/daemon.log.1

Nov 26 06:25:46 log systemd[1]: Reloading The Apache HTTP Server.
Nov 26 06:25:48 log apachectl[11749]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Nov 26 06:25:48 log systemd[1]: Reloaded The Apache HTTP Server.
Nov 26 06:25:59 log fail2ban.server[30559]: INFO flush performed on SYSLOG
Nov 26 06:26:03 log systemd[1]: icinga2.service: Main process exited, code=killed, status=11/SEGV
Nov 26 06:26:03 log systemd[1]: icinga2.service: Unit entered failed state.
Nov 26 06:26:03 log systemd[1]: icinga2.service: Failed with result 'signal'.

File /var/log/icinga2/icinga2.log.1

[2018-11-26 06:24:37 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-26 06:24:43 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-26 06:24:44 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-26 06:25:13 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-26 06:25:14 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-26 06:25:23 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-26 06:25:43 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-11-26 06:25:43 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-11-26 06:25:43 +0100] information/WorkQueue: #15 (JsonRpcConnection, #6) items: 0, rate:  0/s (0/min 0/5min 0/15min);

File /var/log/icinga2/crash/report.1541741165.911804

  Application version: r2.10.1-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-7-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 1880c6977367

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7f20779d7fff]
    (1) libc.so.6: abort (+0x16a) [0x7f20779d942a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7f2077a15c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7f2077a1bfc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7f2077a1c80e]
    (5) icinga2: std::_Function_base::_Base_manager<std::_Bind<std::_Mem_fn<void (icinga::Timer::*)()> (boost::intrusive_ptr<icinga::Timer>)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (+0x38) [0x55a98c8d6358]
    (6) icinga2: icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (+0x4ca) [0x55a98c8c716a]
    (7) libboost_thread.so.1.62.0: <unknown function> (+0x12116) [0x7f207a331116]
    (8) libpthread.so.0: <unknown function> (+0x7494) [0x7f2079353494]
    (9) libc.so.6: clone (+0x3f) [0x7f2077a8dacf]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

File /var/log/messages.1

Nov 26 06:26:01 log kernel: [5674078.627855] traps: icinga2[4528] general protection ip:7f5712a4025b sp:7ffcaa2ff590 error:0
Nov 26 06:26:01 log kernel: [5674078.627860]  in libc-2.24.so[7f57129c9000+195000]
Nov 26 06:26:05 log liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="11297" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

File /var/log/syslog.1

Nov 26 06:26:01 log kernel: [5674078.627855] traps: icinga2[4528] general protection ip:7f5712a4025b sp:7ffcaa2ff590 error:0
Nov 26 06:26:01 log kernel: [5674078.627860]  in libc-2.24.so[7f57129c9000+195000]
Nov 26 06:26:03 log systemd[1]: icinga2.service: Main process exited, code=killed, status=11/SEGV
Nov 26 06:26:03 log systemd[1]: icinga2.service: Unit entered failed state.
Nov 26 06:26:03 log systemd[1]: icinga2.service: Failed with result 'signal'.
Nov 26 06:26:05 log liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="11297" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

File /var/log/apt/history.log

Start-Date: 2018-11-20  11:14:37
Commandline: apt full-upgrade
Requested-By: user (1000)
Upgrade: python3-josepy:amd64 (1.0.1-1~bpo9+1, 1.1.0-2~bpo9+1), icinga2-bin:amd64 (2.10.1-1.stretch, 2.10.2-1.stretch), certbot:amd64 (0.25.0-1~bpo9+1, 0.28.0-1~bpo9+1), icinga2-doc:amd64 (2.10.1-1.stretch, 2.10.2-1.stretch), python3-acme:amd64 (0.25.1-1~bpo9+1, 0.28.0-1~bpo9+1), python3-certbot-apache:amd64 (0.25.0-2~bpo9+1, 0.28.0-1~bpo9+1), icinga2-common:amd64 (2.10.1-1.stretch, 2.10.2-1.stretch), libmariadbclient18:amd64 (10.1.26-0+deb9u1, 10.1.37-0+deb9u1), python3-certbot:amd64 (0.25.0-1~bpo9+1, 0.28.0-1~bpo9+1), python-certbot-apache:amd64 (0.25.0-2~bpo9+1, 0.28.0-1~bpo9+1), icinga2:amd64 (2.10.1-1.stretch, 2.10.2-1.stretch)
End-Date: 2018-11-20  11:16:08
dnsmichi commented 5 years ago

Does 2.10.2 change something here?

fladna9 commented 5 years ago

@dnsmichi : Doesn't seem so, because it crashed this morning.

How to reproduce: good question. Installed Debian Stretch, followed the GrayLog setup tutorial. Nothing really special. As told many times before, i manage +30 Debian/Ubuntu machines, and this one is the only one having this problem. All have latest Stable version (as many are production servers).

dnsmichi commented 5 years ago

Application version: r2.10.1-1 is older than 2.10.2.

fladna9 commented 5 years ago

Oh, it seems i forgot to restart icinga2 service after update.

Since it crashed this morning, i'll have to wait for it to maybe crash next week.

sschindel commented 5 years ago

Ahh, indeed it crashed with an older version. Seems like the package upgrade did not come with a daemon restart hook. I restarted all icinga instances on all my boxes... I will you give some feedback if this issue still occurs or not.

NielsH commented 5 years ago

We haven't had the issues for a little while (2-3 weeks) but just before the weekend this started again on the same server. In addition, we also had the first server running Debian 8 crashing on this. This was a new server that was installed with 2.10.2-1.jessie Logs show:

cat /var/log/icinga2/error.log 
icinga2: /usr/include/boost/signals2/detail/lwm_pthreads.hpp:60: void boost::signals2::mutex::lock(): Assertion `pthread_mutex_lock(&m_) == 0' failed.
Caught SIGABRT.
Current time: 2018-11-26 06:25:17 +0100

Also crash report:

cat /var/log/icinga2/crash/report.1543209917.836953 
  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 3.16.0-7-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: bc2295102b6c

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0x37) [0x2af59a8e9067]
    (1) libc.so.6: abort (+0x148) [0x2af59a8ea448]
    (2) libc.so.6: <unknown function> (+0x2e266) [0x2af59a8e2266]
    (3) libc.so.6: <unknown function> (+0x2e312) [0x2af59a8e2312]
    (4) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x581c9a]
    (5) icinga2: icinga::Timer::~Timer() (+0x112) [0x5f3bb2]
    (6) icinga2: icinga::Timer::~Timer() (+0x9) [0x5f3bd9]
    (7) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa9) [0x5fd399]
    (8) icinga2: icinga::FileLogger::ReopenLogFile() (+0x134) [0x66db24]
    (9) icinga2: icinga::Application::RunEventLoop() (+0x482) [0x5cc632]
    (10) icinga2: icinga::IcingaApplication::Main() (+0x261) [0x7fdfe1]
    (11) icinga2: icinga::Application::Run() (+0xb4) [0x5c7af4]
    (12) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::string, std::allocator<std::string> > const&) const (+0xebc) [0x7d91fc]
    (13) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x5d4d66]
    (14) icinga2: main (+0x47) [0x5830a7]
    (15) libc.so.6: __libc_start_main (+0xf5) [0x2af59a8d5b45]
    (16) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x5904f7]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

I have now installed gdb on the server so if it crashes again I can sent a gdb report.

Also, regarding the Debian 9 server that I also reported earlier - I do think something changed with the new version, but I'm a bit reluctant reporting it since I am not 100% sure and don't wanna give false signals. What I noticed here is that when it crashed, it was disconnected from the Icinga monitoring system (endpoints not connected), but the daemon was still running. I thought it crashed again so I restarted the daemon but only later saw that the daemon was still running. It refused to restart and after a systemd timeout it sent a SIGKILL to the Icinga Daemon and restarted it properly.

I was a bit too soon restarting here on Debian 9, otherwise I'd have investigated more. I thought it was not running when I gave the restart command, but it was... so it might've been possible to get more debug information from it. If it happens again I'll try to get more information. it might also have been a one-time-thing since previously it has always completely crashed, and not keep running but simply disconnected.

sschindel commented 5 years ago

We have still the same issue:

# cat /var/log/icinga2/crash/report.1543469122.888419
  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-6-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 486e413fb159

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7fec4a6a0fff]
    (1) libc.so.6: abort (+0x16a) [0x7fec4a6a242a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7fec4a6dec00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7fec4a6e4fc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7fec4a6e580e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa4) [0x55e06f43c1b4]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x55e06f3f145a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x55e06f3ce8f2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x55e06f5d7b64]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x55e06f3cc8a1]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x55e06f5a7c19]
    (11) icinga2: <unknown function> (+0x266446) [0x55e06f3d2446]
    (12) icinga2: main (+0x49) [0x55e06f376889]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7fec4a68e2e1]
    (14) icinga2: _start (+0x2a) [0x55e06f393d6a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory
dnsmichi commented 5 years ago

Ok, thanks. So this problem only affects Debian 9 Stretch thus far. I would expect that everyone reporting these problems is using Systemd, right?

NielsH commented 5 years ago

@dnsmichi So-far I've also seen it on a single Debian 8 machine; see: https://github.com/Icinga/icinga2/issues/6737#issuecomment-441616058

dnsmichi commented 5 years ago

Ok. Again, with Systemd or not?

NielsH commented 5 years ago

Yes, systemd. That server was a new install with no modifications to the init system.

fladna9 commented 5 years ago

Systemd as well, as it is included as default in Debian.

sschindel commented 5 years ago

Systemd here as well.

NielsH commented 5 years ago

Just had another crash on Debian 9, previously I mentioned I thought the daemon kept running on Debian 9 but no longer connected to the endpoint, however this is not the case now.

Please see the crash report:

cat crash/report.1543814713.893191
  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-8-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 486e413fb159

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7f9cbf3b5fff]
    (1) libc.so.6: abort (+0x16a) [0x7f9cbf3b742a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7f9cbf3f3c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7f9cbf3f9fc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7f9cbf3fa80e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa4) [0x5642d47aa1b4]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x5642d475f45a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x5642d473c8f2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x5642d4945b64]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x5642d473a8a1]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x5642d4915c19]
    (11) icinga2: <unknown function> (+0x266446) [0x5642d4740446]
    (12) icinga2: main (+0x49) [0x5642d46e4889]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7f9cbf3a32e1]
    (14) icinga2: _start (+0x2a) [0x5642d4701d6a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

[New LWP 28461]
[New LWP 28462]
[New LWP 28463]
[New LWP 28464]
[New LWP 28465]
[New LWP 28473]
[New LWP 28478]
[New LWP 28479]
[New LWP 28514]
[New LWP 28516]
[New LWP 28517]
[New LWP 28518]
[New LWP 28519]
[New LWP 28520]
[New LWP 28521]
[New LWP 28522]
[New LWP 28523]
[New LWP 28524]
[New LWP 28525]
[New LWP 28526]
[New LWP 28527]
[New LWP 28528]
[New LWP 28529]
[New LWP 28530]
[New LWP 28531]
[New LWP 28542]
[New LWP 28599]
[New LWP 28600]
[New LWP 28601]
[New LWP 28602]
[New LWP 29089]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f9cc0d3ab3a in __waitpid (pid=29734, stat_loc=0x7fff89dd787c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29  ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.

Thread 32 (Thread 0x7f9cbb3aa700 (LWP 29089)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb3aa700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb3aa700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311132808960, 9004287024282259819, 0, 140311133337375, 140311132808960, 140310676981584, -8949548412424892053, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 31 (Thread 0x7f9cbadd3700 (LWP 28602)):
#0  0x00007f9cbf46267d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47726fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbadd3700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbadd3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311126685440, 9004287024282259819, 0, 140311132794223, 0, 140311249866816, -8949546440498032277, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 30 (Thread 0x7f9cbae14700 (LWP 28601)):
#0  0x00007f9cbf46267d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47726fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbae14700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbae14700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311126951680, 9004287024282259819, 0, 140311132794223, 0, 140311249866816, -8949546817918283413, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 29 (Thread 0x7f9cbae55700 (LWP 28600)):
#0  0x00007f9cbf46267d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47726fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbae55700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbae55700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311127217920, 9004287024282259819, 0, 140311132794223, 0, 140311249866816, -8949546787316641429, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 28 (Thread 0x7f9cbae96700 (LWP 28599)):
#0  0x00007f9cbf46267d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47726fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbae96700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbae96700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311127484160, 9004287024282259819, 0, 140311132794223, 0, 140311249866816, -8949546889858985621, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 27 (Thread 0x7f9cbaed7700 (LWP 28542)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d47fb718 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x00005642d47fc4cb in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x00005642d47c5fce in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x00005642d47cd565 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbaed7700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbaed7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311127750400, 9004287024282259819, 0, 140311131464895, 0, 140311249866816, -8949546854962376341, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 26 (Thread 0x7f9cbaf18700 (LWP 28531)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47cd79b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbaf18700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbaf18700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311128016640, 9004287024282259819, 0, 140311132796927, 0, 140311249866816, -8949546682626813589, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 25 (Thread 0x7f9cbaf59700 (LWP 28530)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47cd79b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbaf59700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbaf59700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311128282880, 9004287024282259819, 0, 140311132796927, 0, 140311249866816, -8949546652025171605, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 24 (Thread 0x7f9cbaf9a700 (LWP 28529)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47cd79b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbaf9a700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbaf9a700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311128549120, 9004287024282259819, 0, 140311132796927, 0, 140311249866816, -8949546754567515797, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 23 (Thread 0x7f9cbafdb700 (LWP 28528)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47cd79b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbafdb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbafdb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311128815360, 9004287024282259819, 0, 140311132796927, 0, 140311249866816, -8949546719670906517, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 22 (Thread 0x7f9cbb01c700 (LWP 28527)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d47fb718 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x00005642d47fc4cb in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x00005642d47c5fce in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x00005642d47cd565 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb01c700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb01c700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311129081600, 9004287024282259819, 0, 140311132796095, 0, 140311249866816, -8949548196602785429, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 21 (Thread 0x7f9cbb05d700 (LWP 28526)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb05d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb05d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311129347840, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548166001143445, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 20 (Thread 0x7f9cbb09e700 (LWP 28525)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb09e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb09e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311129614080, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548268543487637, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 19 (Thread 0x7f9cbb0df700 (LWP 28524)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb0df700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb0df700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311129880320, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548233646878357, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 18 (Thread 0x7f9cbb120700 (LWP 28523)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb120700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb120700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311130146560, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548061311315605, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 17 (Thread 0x7f9cbb161700 (LWP 28522)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb161700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb161700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311130412800, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548030709673621, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 16 (Thread 0x7f9cbb1a2700 (LWP 28521)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb1a2700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb1a2700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311130679040, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548133252017813, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 15 (Thread 0x7f9cbb1e3700 (LWP 28520)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb1e3700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb1e3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311130945280, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548098355408533, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 14 (Thread 0x7f9cbb224700 (LWP 28519)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb224700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb224700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311131211520, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548475775659669, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 13 (Thread 0x7f9cbb265700 (LWP 28518)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb265700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb265700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311131477760, 9004287024282259819, 0, 140311133337375, 0, 140311249866816, -8949548445174017685, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 12 (Thread 0x7f9cbb2a6700 (LWP 28517)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d47a4126 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb2a6700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb2a6700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311131744000, 9004287024282259819, 0, 140311133337903, 0, 140311249866816, -8949548547716361877, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 11 (Thread 0x7f9cbb2e7700 (LWP 28516)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb2e7700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb2e7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311132010240, 9004287024282259819, 0, 140311133337263, 0, 140311249866816, -8949548512819752597, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 10 (Thread 0x7f9cbb369700 (LWP 28514)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb369700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb369700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311132542720, 9004287024282259819, 0, 140311133337263, 0, 140311249866816, -8949548309882547861, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 9 (Thread 0x7f9cbb3eb700 (LWP 28479)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d4a4ea4b in icinga::CheckerComponent::CheckThreadProc() ()
No symbol table info available.
#3  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb3eb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb3eb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311133075200, 9004287024282259819, 0, 140735506385967, 140311133075200, 94845075171552, -8949548377528282773, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 8 (Thread 0x7f9cbb42c700 (LWP 28478)):
#0  0x00007f9cc0d3a3ed in accept () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d479db77 in icinga::Socket::Accept() ()
No symbol table info available.
#2  0x00005642d489c0c7 in icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr<icinga::Socket> const&) ()
No symbol table info available.
#3  0x00005642d48fdb96 in ?? ()
No symbol table info available.
#4  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#5  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb42c700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb42c700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311133341440, 9004287024282259819, 0, 140735506384063, 140311133341440, 94845075098336, -8949547655436906133, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 7 (Thread 0x7f9cbb46d700 (LWP 28473)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
No locals.
#1  0x00005642d47a5453 in icinga::Timer::TimerThreadProc() ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb46d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb46d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311133607680, 9004287024282259819, 0, 140735506386511, 0, 140311249866816, -8949547624835264149, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 6 (Thread 0x7f9cc219b700 (LWP 28465)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d47a4126 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc219b700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc219b700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311248090880, 9004287024282259819, 0, 140735506406639, 0, 140311249866816, -8949739443443412629, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 5 (Thread 0x7f9cc21dc700 (LWP 28464)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc21dc700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc21dc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311248357120, 9004287024282259819, 0, 140735506405999, 0, 140311249866816, -8949739408546803349, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 4 (Thread 0x7f9cc229f700 (LWP 28463)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d4a4d71e in icinga::CheckerComponent::ResultTimerHandler() ()
No symbol table info available.
#5  0x00005642d47c29c1 in ?? ()
No symbol table info available.
#6  0x00005642d4781e74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x00005642d47a6cf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc229f700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc229f700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311249155840, 9004287024282259819, 0, 140735506405999, 140311249155840, 94845069307328, -8949739857907756693, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 3 (Thread 0x7f9cc225e700 (LWP 28462)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d48895d2 in icinga::ApiListener::ApiReconnectTimerHandler() ()
No symbol table info available.
#5  0x00005642d47c29c1 in ?? ()
No symbol table info available.
#6  0x00005642d4781e74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x00005642d47a6cf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc225e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc225e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311248889600, 9004287024282259819, 0, 140735506405999, 140311248889600, 94845069379488, -8949739755365412501, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 2 (Thread 0x7f9cc221d700 (LWP 28461)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d471f5a9 in icinga::ConfigObject::SetAuthority(bool) ()
No symbol table info available.
#4  0x00005642d48964dc in icinga::ApiListener::UpdateObjectAuthority() ()
No symbol table info available.
#5  0x00005642d47c29c1 in ?? ()
No symbol table info available.
#6  0x00005642d4781e74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x00005642d47a6cf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc221d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc221d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311248623360, 9004287024282259819, 0, 140735506405999, 140311248623360, 94845069380400, -8949739790262021781, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 1 (Thread 0x7f9cc22ab480 (LWP 28417)):
#0  0x00007f9cc0d3ab3a in __waitpid (pid=29734, stat_loc=0x7fff89dd787c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
        resultvar = 18446744073709551104
        sc_cancel_oldtype = 0
#1  0x00005642d47167de in icinga::Application::AttachDebugger(icinga::String const&, bool) ()
No symbol table info available.
#2  0x00005642d473aef7 in icinga::Application::SigAbrtHandler(int) ()
No symbol table info available.
#3  <signal handler called>
No locals.
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {0, 2314885530818453536, 8660248813382869024, 7596496373740942904, 3419760881481315694, 7310595060593224044, 3328211180835464301, 4136053824683915057, 3472385483767702371, 3630854822769929520, 8223625903107027763, 3472328295963438455, 4192904167887482928, 7365366848950382640, 3487249781450040121, 7161630943624376368}}
        pid = <optimized out>
        tid = <optimized out>
#5  0x00007f9cbf3b742a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x756e672d78756e69, sa_sigaction = 0x756e672d78756e69}, sa_mask = {__val = {3761119431852583983, 7149858270011683630, 3472328523928253027, 3689120322585769773, 8606977229197436468, 3472328296226648109, 3475143045726351408, 7378645556122361904, 3472387885476952166, 4123158048845278512, 8223625903103567204, 3472328295963438455, 4192904167887482928, 2314885531086893104, 2314885530818453536, 140735506385568}}, sa_flags = 123, sa_restorer = 0x7fff89dd8aa0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#6  0x00007f9cbf3f3c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f9cbf4e8d98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
        ap = {{gp_offset = 40, fp_offset = 32668, overflow_arg_area = 0x7fff89dd8ab0, reg_save_area = 0x7fff89dd8a40}}
        fd = 2
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#7  0x00007f9cbf3f9fc6 in malloc_printerr (action=3, str=0x7f9cbf4e8ef0 "double free or corruption (fasttop)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
        buf = "00005642d7013a70"
        cp = <optimized out>
        ar_ptr = <optimized out>
        ptr = <optimized out>
        str = 0x7f9cbf4e8ef0 "double free or corruption (fasttop)"
        action = 3
#8  0x00007f9cbf3fa80e in _int_free (av=0x7f9cbf71cb00 <main_arena>, p=0x5642d7013a60, have_lock=0) at malloc.c:3905
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = <optimized out>
        locked = <optimized out>
        __func__ = "_int_free"
#9  0x00005642d47aa1b4 in icinga::StreamLogger::BindStream(std::ostream*, bool) ()
No symbol table info available.
#10 0x00005642d475f45a in icinga::FileLogger::ReopenLogFile() ()
No symbol table info available.
#11 0x00005642d473c8f2 in icinga::Application::RunEventLoop() ()
No symbol table info available.
#12 0x00005642d4945b64 in icinga::IcingaApplication::Main() ()
No symbol table info available.
#13 0x00005642d473a8a1 in icinga::Application::Run() ()
No symbol table info available.
#14 0x00005642d4915c19 in icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const ()
No symbol table info available.
#15 0x00005642d4740446 in ?? ()
No symbol table info available.
#16 0x00005642d46e4889 in main ()
No symbol table info available.
fladna9 commented 5 years ago

Update: happened again on the same machine as usual. I installed gdb for more crash output to help out. Hope it'll help you guys :)

Reminder: Debian Stretch up-to-date, using systemd, Stable Icinga2 repository, package version r2.10.2-1

/var/log/icinga2/crash/report.1543728730.982800

  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-7-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 486e413fb159

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7f3b5c9f6fff]
    (1) libc.so.6: abort (+0x16a) [0x7f3b5c9f842a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7f3b5ca34c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7f3b5ca3afc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7f3b5ca3b80e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa4) [0x55c0505031b4]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x55c0504b845a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x55c0504958f2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x55c05069eb64]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x55c0504938a1]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x55c05066ec19]
    (11) icinga2: <unknown function> (+0x266446) [0x55c050499446]
    (12) icinga2: main (+0x49) [0x55c05043d889]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7f3b5c9e42e1]
    (14) icinga2: _start (+0x2a) [0x55c05045ad6a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

[New LWP 24901]
[New LWP 24902]
[New LWP 24904]
[New LWP 24905]
[New LWP 24913]
[New LWP 24931]
[New LWP 24932]
[New LWP 24933]
[New LWP 24935]
[New LWP 24936]
[New LWP 24937]
[New LWP 24938]
[New LWP 24939]
[New LWP 24940]
[New LWP 24941]
[New LWP 24942]
[New LWP 24943]
[New LWP 24944]
[New LWP 24945]
[New LWP 24946]
[New LWP 24947]
[New LWP 24948]
[New LWP 24949]
[New LWP 24950]
[New LWP 24951]
[New LWP 25471]
[New LWP 25472]
[New LWP 25473]
[New LWP 25474]
[New LWP 21531]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f3b5e37bb3a in __waitpid (pid=21230, stat_loc=0x7ffefe45533c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29  ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.

Thread 31 (Thread 0x7f3ae3fbe700 (LWP 21531)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115
        id = 1580707676
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x000055c0504785a9 in icinga::ConfigObject::SetAuthority(bool) ()
No symbol table info available.
#4  0x000055c0505ef4dc in icinga::ApiListener::UpdateObjectAuthority() ()
No symbol table info available.
#5  0x000055c05051b9c1 in ?? ()
No symbol table info available.
#6  0x000055c0504dae74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x000055c0504ffcf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f3b5e372494 in start_thread (arg=0x7f3ae3fbe700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3ae3fbe700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139890909767424, 1733757316368709993, 0, 139892981867343, 0, 139892983652416, -1839072971467769495, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 30 (Thread 0x7f3ae3fff700 (LWP 25474)):
#0  0x00007f3b5caa367d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504cb6fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3ae3fff700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3ae3fff700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139890910033664, 1733757316368709993, 0, 139892866313583, 0, 139892983652416, -1839072937644902039, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 29 (Thread 0x7f3b58079700 (LWP 25473)):
#0  0x00007f3b5caa367d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504cb6fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b58079700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58079700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892856690432, 1733757316368709993, 0, 139892866313583, 0, 139892983652416, -1839373994451256983, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 28 (Thread 0x7f3b580ba700 (LWP 25472)):
#0  0x00007f3b5caa367d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504cb6fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b580ba700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b580ba700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892856956672, 1733757316368709993, 0, 139892866313583, 0, 139892983652416, -1839374098067342999, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 27 (Thread 0x7f3b580fb700 (LWP 25471)):
#0  0x00007f3b5caa367d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504cb6fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b580fb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b580fb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892857222912, 1733757316368709993, 0, 139892866313583, 0, 139892983652416, -1839374064244475543, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 26 (Thread 0x7f3b5813c700 (LWP 24951)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115
        id = 1580707676
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x000055c0504b58d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x000055c050554718 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x000055c0505554cb in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x000055c05051efce in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x000055c050526565 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f3b5e372494 in start_thread (arg=0x7f3b5813c700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b5813c700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892857489152, 1733757316368709993, 0, 139892860609807, 0, 139892983652416, -1839373892982654615, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 25 (Thread 0x7f3b5817d700 (LWP 24950)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c05052679b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b5817d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b5817d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892857755392, 1733757316368709993, 0, 139892865249151, 0, 139892983652416, -1839373859159787159, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 24 (Thread 0x7f3b581be700 (LWP 24949)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c05052679b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b581be700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b581be700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892858021632, 1733757316368709993, 0, 139892865249151, 0, 139892983652416, -1839373962775873175, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 23 (Thread 0x7f3b581ff700 (LWP 24948)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c05052679b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b581ff700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b581ff700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892858287872, 1733757316368709993, 0, 139892865249151, 0, 139892983652416, -1839373928953005719, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 22 (Thread 0x7f3b58240700 (LWP 24947)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c05052679b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b58240700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58240700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892858554112, 1733757316368709993, 0, 139892865249151, 0, 139892983652416, -1839373757691184791, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 21 (Thread 0x7f3b58281700 (LWP 24946)):
#0  0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b58281700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58281700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892858820352, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373792587794071, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 20 (Thread 0x7f3b582c2700 (LWP 24945)):
#0  0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b582c2700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b582c2700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892859086592, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373758764926615, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 19 (Thread 0x7f3b58303700 (LWP 24944)):
#0  0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b58303700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58303700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892859352832, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373587503105687, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 18 (Thread 0x7f3b58344700 (LWP 24943)):
#0  0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b58344700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58344700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892859619072, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373553680238231, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 17 (Thread 0x7f3b58385700 (LWP 24942)):
#0  0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b58385700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58385700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892859885312, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373657296324247, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 16 (Thread 0x7f3b583c6700 (LWP 24941)):
#0  0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b583c6700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b583c6700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892860151552, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373623473456791, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 15 (Thread 0x7f3b58407700 (LWP 24940)):
#0  0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b58407700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58407700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892860417792, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373452211635863, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 14 (Thread 0x7f3b58448700 (LWP 24939)):
#0  0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b58448700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58448700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892860684032, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373418388768407, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 13 (Thread 0x7f3b588a6700 (LWP 24938)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b588a6700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b588a6700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892865263360, 1733757316368709993, 0, 139892873056479, 139892865263360, 94284516776112, -1839375186841552535, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 12 (Thread 0x7f3b588e7700 (LWP 24937)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115
        id = 1580707676
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x000055c0504b58d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x000055c0504fd126 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007f3b5e372494 in start_thread (arg=0x7f3b588e7700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b588e7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892865529600, 1733757316368709993, 0, 139892873057007, 139892865529600, 94284516737424, -1839375153018685079, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 11 (Thread 0x7f3b58928700 (LWP 24936)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b58928700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58928700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892865795840, 1733757316368709993, 0, 139892873056367, 139892865795840, 94284516775776, -1839374981756864151, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 10 (Thread 0x7f3b58969700 (LWP 24935)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b58969700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b58969700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892866062080, 1733757316368709993, 0, 139892873056367, 139892866062080, 94284516774960, -1839374947933996695, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 9 (Thread 0x7f3b5f8df700 (LWP 24933)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b5f8df700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b5f8df700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892982937344, 1733757316368709993, 0, 139892873056367, 139892982937344, 94284511321424, -1839381745793484439, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 8 (Thread 0x7f3b5f89e700 (LWP 24932)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c0507a7a4b in icinga::CheckerComponent::CheckThreadProc() ()
No symbol table info available.
#3  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b5f89e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b5f89e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892982671104, 1733757316368709993, 0, 140733164381599, 139892982671104, 94284511393584, -1839381779616351895, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 7 (Thread 0x7f3b5f85d700 (LWP 24931)):
#0  0x00007f3b5e37b3ed in accept () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x000055c0504f6b77 in icinga::Socket::Accept() ()
No symbol table info available.
#2  0x000055c0505f50c7 in icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr<icinga::Socket> const&) ()
No symbol table info available.
#3  0x000055c050656b96 in ?? ()
No symbol table info available.
#4  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#5  0x00007f3b5e372494 in start_thread (arg=0x7f3b5f85d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b5f85d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892982404864, 1733757316368709993, 0, 140733164379695, 139892982404864, 94284511394496, -1839381676000265879, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 6 (Thread 0x7f3b5f81c700 (LWP 24913)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
No locals.
#1  0x000055c0504fe453 in icinga::Timer::TimerThreadProc() ()
No symbol table info available.
#2  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f3b5e372494 in start_thread (arg=0x7f3b5f81c700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b5f81c700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892982138624, 1733757316368709993, 0, 140733164382143, 139892982138624, 94284511404816, -1839381709823133335, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 5 (Thread 0x7f3b5f7db700 (LWP 24905)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115
        id = 1580707676
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x000055c0504b58d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x000055c0504fd126 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007f3b5e372494 in start_thread (arg=0x7f3b5f7db700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b5f7db700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892981872384, 1733757316368709993, 0, 140733164402271, 139892981872384, 94284511405728, -1839379682061698711, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 4 (Thread 0x7f3b590da700 (LWP 24904)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f3b5e372494 in start_thread (arg=0x7f3b590da700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b590da700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892873864960, 1733757316368709993, 0, 140733164401631, 139892873864960, 94284511406640, -1839376245550990999, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 3 (Thread 0x7f3b59058700 (LWP 24902)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115
        id = 1580707676
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x000055c0504b58d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x000055c0507a671e in icinga::CheckerComponent::ResultTimerHandler() ()
No symbol table info available.
#5  0x000055c05051b9c1 in ?? ()
No symbol table info available.
#6  0x000055c0504dae74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x000055c0504ffcf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f3b5e372494 in start_thread (arg=0x7f3b59058700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b59058700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892873332480, 1733757316368709993, 0, 140733164401631, 139892873332480, 94284511356432, -1839376175757772439, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 2 (Thread 0x7f3b59017700 (LWP 24901)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115
        id = 1580707676
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x000055c0504b58d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x000055c0505e278e in icinga::ApiListener::ApiReconnectTimerHandler() ()
No symbol table info available.
#5  0x000055c05051b9c1 in ?? ()
No symbol table info available.
#6  0x000055c0504dae74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x000055c0504ffcf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f3b5e372494 in start_thread (arg=0x7f3b59017700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f3b59017700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892873066240, 1733757316368709993, 0, 140733164401631, 139892873066240, 94284511398640, -1839376209580639895, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 1 (Thread 0x7f3b5f8eb480 (LWP 24865)):
#0  0x00007f3b5e37bb3a in __waitpid (pid=21230, stat_loc=0x7ffefe45533c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
        resultvar = 18446744073709551104
        sc_cancel_oldtype = 0
#1  0x000055c05046f7de in icinga::Application::AttachDebugger(icinga::String const&, bool) ()
No symbol table info available.
#2  0x000055c050493ef7 in icinga::Application::SigAbrtHandler(int) ()
No symbol table info available.
#3  <signal handler called>
No locals.
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {0, 3472339291342909488, 7306639832034324512, 3472328511026652518, 3775536448315733805, 8606977229197435701, 3472328296226648109, 3475143045726351408, 2314885530819502128, 2314885530818453536, 2314885530818453536, 8391150724257423392, 7378645557149655905, 3472335116723119717, 7306638935931170096, 8223625903104091189}}
        pid = <optimized out>
        tid = <optimized out>
#5  0x00007f3b5c9f842a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x623366370a302e37, sa_sigaction = 0x623366370a302e37}, sa_mask = {__val = {3472328335000430133, 4136052003671783213, 3274715270390768439, 3762810472192045101, 3475151841819373616, 4121131437673029681, 2314885530818453558, 2314885530818453536, 7596498486491619360, 3762299225194901346, 7434731005617466413, 8751446705837143406, 8030813569398699123, 734137553844580398, 3979324125225379383, 140733164381200}}, sa_flags = 123, sa_restorer = 0x7ffefe456410}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#6  0x00007f3b5ca34c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f3b5cb29d98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
        ap = {{gp_offset = 40, fp_offset = 32571, overflow_arg_area = 0x7ffefe456420, reg_save_area = 0x7ffefe4563b0}}
        fd = 2
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#7  0x00007f3b5ca3afc6 in malloc_printerr (action=3, str=0x7f3b5cb29ef0 "double free or corruption (fasttop)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
        buf = "000055c052fe14f0"
        cp = <optimized out>
        ar_ptr = <optimized out>
        ptr = <optimized out>
        str = 0x7f3b5cb29ef0 "double free or corruption (fasttop)"
        action = 3
#8  0x00007f3b5ca3b80e in _int_free (av=0x7f3b5cd5db00 <main_arena>, p=0x55c052fe14e0, have_lock=0) at malloc.c:3905
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = <optimized out>
        locked = <optimized out>
        __func__ = "_int_free"
#9  0x000055c0505031b4 in icinga::StreamLogger::BindStream(std::ostream*, bool) ()
No symbol table info available.
#10 0x000055c0504b845a in icinga::FileLogger::ReopenLogFile() ()
No symbol table info available.
#11 0x000055c0504958f2 in icinga::Application::RunEventLoop() ()
No symbol table info available.
#12 0x000055c05069eb64 in icinga::IcingaApplication::Main() ()
No symbol table info available.
#13 0x000055c0504938a1 in icinga::Application::Run() ()
No symbol table info available.
#14 0x000055c05066ec19 in icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const ()
No symbol table info available.
#15 0x000055c050499446 in ?? ()
No symbol table info available.
#16 0x000055c05043d889 in main ()
No symbol table info available.

/var/log/icinga2/error.log.1

*** Error in `/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2': double free or corruption (fasttop): 0x000055c052fe14f0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7f3b5ca34bfb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fc6)[0x7f3b5ca3afc6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7780e)[0x7f3b5ca3b80e]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga12StreamLogger10BindStreamEPSob+0xa4)[0x55c0505031b4]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga10FileLogger13ReopenLogFileEv+0xaa)[0x55c0504b845a]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga11Application12RunEventLoopEv+0x7c2)[0x55c0504958f2]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga17IcingaApplication4MainEv+0x1f4)[0x55c05069eb64]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga11Application3RunEv+0xc1)[0x55c0504938a1]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZNK6icinga13DaemonCommand3RunERKN5boost15program_options13variables_mapERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISC_EE+0xbc9)[0x55c05066ec19]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(+0x266446)[0x55c050499446]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(main+0x49)[0x55c05043d889]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f3b5c9e42e1]
/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_start+0x2a)[0x55c05045ad6a]
======= Memory map: ========
55c050233000-55c050b59000 r-xp 00000000 08:01 10355102                   /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
55c050d59000-55c050d8d000 r--p 00926000 08:01 10355102                   /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
55c050d8d000-55c050d8f000 rw-p 0095a000 08:01 10355102                   /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
55c050d8f000-55c050d96000 rw-p 00000000 00:00 0
55c052c7b000-55c053242000 rw-p 00000000 00:00 0                          [heap]
7f3ad4000000-7f3ad4021000 rw-p 00000000 00:00 0
7f3ad4021000-7f3ad8000000 ---p 00000000 00:00 0
7f3ad8000000-7f3ad8021000 rw-p 00000000 00:00 0
7f3ad8021000-7f3adc000000 ---p 00000000 00:00 0
7f3adc000000-7f3adc021000 rw-p 00000000 00:00 0
7f3adc021000-7f3ae0000000 ---p 00000000 00:00 0
7f3ae3f7e000-7f3ae3f7f000 ---p 00000000 00:00 0
7f3ae3f7f000-7f3ae3fbf000 rw-p 00000000 00:00 0
7f3ae3fbf000-7f3ae3fc0000 ---p 00000000 00:00 0
7f3ae3fc0000-7f3ae4000000 rw-p 00000000 00:00 0
7f3ae4000000-7f3ae4021000 rw-p 00000000 00:00 0
7f3ae4021000-7f3ae8000000 ---p 00000000 00:00 0
7f3ae8000000-7f3ae8021000 rw-p 00000000 00:00 0
7f3ae8021000-7f3aec000000 ---p 00000000 00:00 0
7f3aec000000-7f3aec021000 rw-p 00000000 00:00 0
7f3aec021000-7f3af0000000 ---p 00000000 00:00 0
7f3af0000000-7f3af0021000 rw-p 00000000 00:00 0
7f3af0021000-7f3af4000000 ---p 00000000 00:00 0
7f3af4000000-7f3af4021000 rw-p 00000000 00:00 0
7f3af4021000-7f3af8000000 ---p 00000000 00:00 0
7f3af8000000-7f3af8021000 rw-p 00000000 00:00 0
7f3af8021000-7f3afc000000 ---p 00000000 00:00 0
7f3afc000000-7f3afc021000 rw-p 00000000 00:00 0
7f3afc021000-7f3b00000000 ---p 00000000 00:00 0
7f3b00000000-7f3b00021000 rw-p 00000000 00:00 0
7f3b00021000-7f3b04000000 ---p 00000000 00:00 0
7f3b04000000-7f3b04021000 rw-p 00000000 00:00 0
7f3b04021000-7f3b08000000 ---p 00000000 00:00 0
7f3b08000000-7f3b08021000 rw-p 00000000 00:00 0
7f3b08021000-7f3b0c000000 ---p 00000000 00:00 0
7f3b0c000000-7f3b0c021000 rw-p 00000000 00:00 0
7f3b0c021000-7f3b10000000 ---p 00000000 00:00 0
7f3b10000000-7f3b10021000 rw-p 00000000 00:00 0
7f3b10021000-7f3b14000000 ---p 00000000 00:00 0
7f3b14000000-7f3b14021000 rw-p 00000000 00:00 0
7f3b14021000-7f3b18000000 ---p 00000000 00:00 0
7f3b18000000-7f3b18021000 rw-p 00000000 00:00 0
7f3b18021000-7f3b1c000000 ---p 00000000 00:00 0
7f3b1c000000-7f3b1c042000 rw-p 00000000 00:00 0
7f3b1c042000-7f3b20000000 ---p 00000000 00:00 0
7f3b20000000-7f3b20021000 rw-p 00000000 00:00 0
7f3b20021000-7f3b24000000 ---p 00000000 00:00 0
7f3b24000000-7f3b2406c000 rw-p 00000000 00:00 0
7f3b2406c000-7f3b28000000 ---p 00000000 00:00 0
7f3b28000000-7f3b2804c000 rw-p 00000000 00:00 0
7f3b2804c000-7f3b2c000000 ---p 00000000 00:00 0
7f3b2c000000-7f3b2c05e000 rw-p 00000000 00:00 0
7f3b2c05e000-7f3b30000000 ---p 00000000 00:00 0
7f3b30000000-7f3b30021000 rw-p 00000000 00:00 0
7f3b30021000-7f3b34000000 ---p 00000000 00:00 0
7f3b34000000-7f3b34056000 rw-p 00000000 00:00 0
7f3b34056000-7f3b38000000 ---p 00000000 00:00 0
7f3b38000000-7f3b38021000 rw-p 00000000 00:00 0
7f3b38021000-7f3b3c000000 ---p 00000000 00:00 0
7f3b3c000000-7f3b3c02b000 rw-p 00000000 00:00 0
7f3b3c02b000-7f3b40000000 ---p 00000000 00:00 0
7f3b40000000-7f3b40021000 rw-p 00000000 00:00 0
7f3b40021000-7f3b44000000 ---p 00000000 00:00 0
7f3b44000000-7f3b44021000 rw-p 00000000 00:00 0
7f3b44021000-7f3b48000000 ---p 00000000 00:00 0
7f3b48000000-7f3b48021000 rw-p 00000000 00:00 0
7f3b48021000-7f3b4c000000 ---p 00000000 00:00 0
7f3b4c000000-7f3b4c076000 rw-p 00000000 00:00 0
7f3b4c076000-7f3b50000000 ---p 00000000 00:00 0
7f3b50000000-7f3b50050000 rw-p 00000000 00:00 0
7f3b50050000-7f3b54000000 ---p 00000000 00:00 0
7f3b54000000-7f3b54054000 rw-p 00000000 00:00 0
7f3b54054000-7f3b58000000 ---p 00000000 00:00 0
7f3b58039000-7f3b5803a000 ---p 00000000 00:00 0
7f3b5803a000-7f3b5807a000 rw-p 00000000 00:00 0
7f3b5807a000-7f3b5807b000 ---p 00000000 00:00 0
7f3b5807b000-7f3b580bb000 rw-p 00000000 00:00 0
7f3b580bb000-7f3b580bc000 ---p 00000000 00:00 0
7f3b580bc000-7f3b580fc000 rw-p 00000000 00:00 0
7f3b580fc000-7f3b580fd000 ---p 00000000 00:00 0
7f3b580fd000-7f3b5813d000 rw-p 00000000 00:00 0
7f3b5813d000-7f3b5813e000 ---p 00000000 00:00 0
7f3b5813e000-7f3b5817e000 rw-p 00000000 00:00 0
7f3b5817e000-7f3b5817f000 ---p 00000000 00:00 0
7f3b5817f000-7f3b581bf000 rw-p 00000000 00:00 0
7f3b581bf000-7f3b581c0000 ---p 00000000 00:00 0
7f3b581c0000-7f3b58200000 rw-p 00000000 00:00 0
7f3b58200000-7f3b58201000 ---p 00000000 00:00 0
7f3b58201000-7f3b58241000 rw-p 00000000 00:00 0
7f3b58241000-7f3b58242000 ---p 00000000 00:00 0
7f3b58242000-7f3b58282000 rw-p 00000000 00:00 0
7f3b58282000-7f3b58283000 ---p 00000000 00:00 0
7f3b58283000-7f3b582c3000 rw-p 00000000 00:00 0
7f3b582c3000-7f3b582c4000 ---p 00000000 00:00 0
7f3b582c4000-7f3b58304000 rw-p 00000000 00:00 0
7f3b58304000-7f3b58305000 ---p 00000000 00:00 0
7f3b58305000-7f3b58345000 rw-p 00000000 00:00 0
7f3b58345000-7f3b58346000 ---p 00000000 00:00 0
7f3b58346000-7f3b58386000 rw-p 00000000 00:00 0
7f3b58386000-7f3b58387000 ---p 00000000 00:00 0
7f3b58387000-7f3b583c7000 rw-p 00000000 00:00 0
7f3b583c7000-7f3b583c8000 ---p 00000000 00:00 0
7f3b583c8000-7f3b58408000 rw-p 00000000 00:00 0
7f3b58408000-7f3b58409000 ---p 00000000 00:00 0
7f3b58409000-7f3b58449000 rw-p 00000000 00:00 0
7f3b58449000-7f3b5845d000 r-xp 00000000 08:01 2752536                    /lib/x86_64-linux-gnu/libresolv-2.24.so
7f3b5845d000-7f3b5865c000 ---p 00014000 08:01 2752536                    /lib/x86_64-linux-gnu/libresolv-2.24.so
7f3b5865c000-7f3b5865d000 r--p 00013000 08:01 2752536                    /lib/x86_64-linux-gnu/libresolv-2.24.so
7f3b5865d000-7f3b5865e000 rw-p 00014000 08:01 2752536                    /lib/x86_64-linux-gnu/libresolv-2.24.so
7f3b5865e000-7f3b58660000 rw-p 00000000 00:00 0
7f3b58660000-7f3b58665000 r-xp 00000000 08:01 2752529                    /lib/x86_64-linux-gnu/libnss_dns-2.24.so
7f3b58665000-7f3b58864000 ---p 00005000 08:01 2752529                    /lib/x86_64-linux-gnu/libnss_dns-2.24.so
7f3b58864000-7f3b58865000 r--p 00004000 08:01 2752529                    /lib/x86_64-linux-gnu/libnss_dns-2.24.so
7f3b58865000-7f3b58866000 rw-p 00005000 08:01 2752529                    /lib/x86_64-linux-gnu/libnss_dns-2.24.so
7f3b58866000-7f3b58867000 ---p 00000000 00:00 0
7f3b58867000-7f3b588a7000 rw-p 00000000 00:00 0
7f3b588a7000-7f3b588a8000 ---p 00000000 00:00 0
7f3b588a8000-7f3b588e8000 rw-p 00000000 00:00 0
7f3b588e8000-7f3b588e9000 ---p 00000000 00:00 0
7f3b588e9000-7f3b58929000 rw-p 00000000 00:00 0
7f3b58929000-7f3b5892a000 ---p 00000000 00:00 0
7f3b5892a000-7f3b5896a000 rw-p 00000000 00:00 0
7f3b5896a000-7f3b5896b000 ---p 00000000 00:00 0
7f3b5896b000-7f3b589ab000 rw-p 00000000 00:00 0
7f3b589ab000-7f3b589b6000 r-xp 00000000 08:01 2752532                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
7f3b589b6000-7f3b58bb5000 ---p 0000b000 08:01 2752532                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
7f3b58bb5000-7f3b58bb6000 r--p 0000a000 08:01 2752532                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
7f3b58bb6000-7f3b58bb7000 rw-p 0000b000 08:01 2752532                    /lib/x86_64-linux-gnu/libnss_nis-2.24.so
7f3b58bb7000-7f3b58bcb000 r-xp 00000000 08:01 2752527                    /lib/x86_64-linux-gnu/libnsl-2.24.so
7f3b58bcb000-7f3b58dcb000 ---p 00014000 08:01 2752527                    /lib/x86_64-linux-gnu/libnsl-2.24.so
7f3b58dcb000-7f3b58dcc000 r--p 00014000 08:01 2752527                    /lib/x86_64-linux-gnu/libnsl-2.24.so
7f3b58dcc000-7f3b58dcd000 rw-p 00015000 08:01 2752527                    /lib/x86_64-linux-gnu/libnsl-2.24.so
7f3b58dcd000-7f3b58dcf000 rw-p 00000000 00:00 0
7f3b58dcf000-7f3b58dd6000 r-xp 00000000 08:01 2752528                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
7f3b58dd6000-7f3b58fd5000 ---p 00007000 08:01 2752528                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
7f3b58fd5000-7f3b58fd6000 r--p 00006000 08:01 2752528                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
7f3b58fd6000-7f3b58fd7000 rw-p 00007000 08:01 2752528                    /lib/x86_64-linux-gnu/libnss_compat-2.24.so
7f3b58fd7000-7f3b58fd8000 ---p 00000000 00:00 0
7f3b58fd8000-7f3b59018000 rw-p 00000000 00:00 0
7f3b59018000-7f3b59019000 ---p 00000000 00:00 0
7f3b59019000-7f3b59059000 rw-p 00000000 00:00 0
7f3b59059000-7f3b5905a000 ---p 00000000 00:00 0
7f3b5905a000-7f3b5909a000 rw-p 00000000 00:00 0
7f3b5909a000-7f3b5909b000 ---p 00000000 00:00 0
7f3b5909b000-7f3b590db000 rw-p 00000000 00:00 0
7f3b590db000-7f3b590e5000 r-xp 00000000 08:01 2752530                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
7f3b590e5000-7f3b592e5000 ---p 0000a000 08:01 2752530                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
7f3b592e5000-7f3b592e6000 r--p 0000a000 08:01 2752530                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
7f3b592e6000-7f3b592e7000 rw-p 0000b000 08:01 2752530                    /lib/x86_64-linux-gnu/libnss_files-2.24.so
7f3b592e7000-7f3b592ed000 rw-p 00000000 00:00 0
7f3b592ed000-7f3b59300000 r-xp 00000000 08:01 2752562                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
7f3b59300000-7f3b594ff000 ---p 00013000 08:01 2752562                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
7f3b594ff000-7f3b59500000 r--p 00012000 08:01 2752562                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
7f3b59500000-7f3b59501000 rw-p 00013000 08:01 2752562                    /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0
7f3b59501000-7f3b59573000 r-xp 00000000 08:01 2752551                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f3b59573000-7f3b59772000 ---p 00072000 08:01 2752551                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f3b59772000-7f3b59773000 r--p 00071000 08:01 2752551                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f3b59773000-7f3b59774000 rw-p 00072000 08:01 2752551                    /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f3b59774000-7f3b59787000 r-xp 00000000 08:01 2753036                    /lib/x86_64-linux-gnu/libbsd.so.0.8.3
7f3b59787000-7f3b59987000 ---p 00013000 08:01 2753036                    /lib/x86_64-linux-gnu/libbsd.so.0.8.3
7f3b59987000-7f3b59988000 r--p 00013000 08:01 2753036                    /lib/x86_64-linux-gnu/libbsd.so.0.8.3
7f3b59988000-7f3b59989000 rw-p 00014000 08:01 2753036                    /lib/x86_64-linux-gnu/libbsd.so.0.8.3
7f3b59989000-7f3b5998a000 rw-p 00000000 00:00 0
7f3b5998a000-7f3b599ac000 r-xp 00000000 08:01 2752566                    /lib/x86_64-linux-gnu/libncurses.so.5.9
7f3b599ac000-7f3b59bab000 ---p 00022000 08:01 2752566                    /lib/x86_64-linux-gnu/libncurses.so.5.9
7f3b59bab000-7f3b59bac000 r--p 00021000 08:01 2752566                    /lib/x86_64-linux-gnu/libncurses.so.5.9
7f3b59bac000-7f3b59bad000 rw-p 00022000 08:01 2752566                    /lib/x86_64-linux-gnu/libncurses.so.5.9
7f3b59bad000-7f3b59cb4000 r-xp 00000000 08:01 2752564                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
7f3b59cb4000-7f3b59eb4000 ---p 00107000 08:01 2752564                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
7f3b59eb4000-7f3b59eb6000 r--p 00107000 08:01 2752564                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
7f3b59eb6000-7f3b59ebd000 rw-p 00109000 08:01 2752564                    /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6
7f3b59ebd000-7f3b59ece000 r-xp 00000000 08:01 10224540                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7f3b59ece000-7f3b5a0cd000 ---p 00011000 08:01 10224540                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7f3b5a0cd000-7f3b5a0ce000 r--p 00010000 08:01 10224540                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7f3b5a0ce000-7f3b5a0cf000 rw-p 00011000 08:01 10224540                   /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1
7f3b5a0cf000-7f3b5a0f4000 r-xp 00000000 08:01 2752545                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
7f3b5a0f4000-7f3b5a2f3000 ---p 00025000 08:01 2752545                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
7f3b5a2f3000-7f3b5a2f4000 r--p 00024000 08:01 2752545                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
7f3b5a2f4000-7f3b5a2f5000 rw-p 00025000 08:01 2752545                    /lib/x86_64-linux-gnu/liblzma.so.5.2.2
7f3b5a2f5000-7f3b5a31a000 r-xp 00000000 08:01 2752601                    /lib/x86_64-linux-gnu/libselinux.so.1
7f3b5a31a000-7f3b5a519000 ---p 00025000 08:01 2752601                    /lib/x86_64-linux-gnu/libselinux.so.1
7f3b5a519000-7f3b5a51a000 r--p 00024000 08:01 2752601                    /lib/x86_64-linux-gnu/libselinux.so.1
7f3b5a51a000-7f3b5a51b000 rw-p 00025000 08:01 2752601                    /lib/x86_64-linux-gnu/libselinux.so.1
7f3b5a51b000-7f3b5a51d000 rw-p 00000000 00:00 0
7f3b5a51d000-7f3b5a6b1000 r-xp 00000000 08:01 10231348                   /usr/lib/x86_64-linux-gnu/libicuuc.so.57.1
7f3b5a6b1000-7f3b5a8b0000 ---p 00194000 08:01 10231348                   /usr/lib/x86_64-linux-gnu/libicuuc.so.57.1
7f3b5a8b0000-7f3b5a8c2000 r--p 00193000 08:01 10231348                   /usr/lib/x86_64-linux-gnu/libicuuc.so.57.1
7f3b5a8c2000-7f3b5a8c3000 rw-p 001a5000 08:01 10231348                   /usr/lib/x86_64-linux-gnu/libicuuc.so.57.1
7f3b5a8c3000-7f3b5a8c5000 rw-p 00000000 00:00 0
7f3b5a8c5000-7f3b5ab30000 r-xp 00000000 08:01 10231342                   /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1
7f3b5ab30000-7f3b5ad2f000 ---p 0026b000 08:01 10231342                   /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1
7f3b5ad2f000-7f3b5ad3c000 r--p 0026a000 08:01 10231342                   /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1
7f3b5ad3c000-7f3b5ad3e000 rw-p 00277000 08:01 10231342                   /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1
7f3b5ad3e000-7f3b5ad3f000 rw-p 00000000 00:00 0
7f3b5ad3f000-7f3b5c5bb000 r-xp 00000000 08:01 10231341                   /usr/lib/x86_64-linux-gnu/libicudata.so.57.1
7f3b5c5bb000-7f3b5c7ba000 ---p 0187c000 08:01 10231341                   /usr/lib/x86_64-linux-gnu/libicudata.so.57.1
7f3b5c7ba000-7f3b5c7bb000 r--p 0187b000 08:01 10231341                   /usr/lib/x86_64-linux-gnu/libicudata.so.57.1
7f3b5c7bb000-7f3b5c7bc000 rw-p 0187c000 08:01 10231341                   /usr/lib/x86_64-linux-gnu/libicudata.so.57.1
7f3b5c7bc000-7f3b5c7c3000 r-xp 00000000 08:01 2752537                    /lib/x86_64-linux-gnu/librt-2.24.so
7f3b5c7c3000-7f3b5c9c2000 ---p 00007000 08:01 2752537                    /lib/x86_64-linux-gnu/librt-2.24.so
7f3b5c9c2000-7f3b5c9c3000 r--p 00006000 08:01 2752537                    /lib/x86_64-linux-gnu/librt-2.24.so
7f3b5c9c3000-7f3b5c9c4000 rw-p 00007000 08:01 2752537                    /lib/x86_64-linux-gnu/librt-2.24.so
7f3b5c9c4000-7f3b5cb59000 r-xp 00000000 08:01 2752520                    /lib/x86_64-linux-gnu/libc-2.24.so
7f3b5cb59000-7f3b5cd59000 ---p 00195000 08:01 2752520                    /lib/x86_64-linux-gnu/libc-2.24.so
7f3b5cd59000-7f3b5cd5d000 r--p 00195000 08:01 2752520                    /lib/x86_64-linux-gnu/libc-2.24.so
7f3b5cd5d000-7f3b5cd5f000 rw-p 00199000 08:01 2752520                    /lib/x86_64-linux-gnu/libc-2.24.so
7f3b5cd5f000-7f3b5cd63000 rw-p 00000000 00:00 0
7f3b5cd63000-7f3b5cd79000 r-xp 00000000 08:01 2752516                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3b5cd79000-7f3b5cf78000 ---p 00016000 08:01 2752516                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3b5cf78000-7f3b5cf79000 r--p 00015000 08:01 2752516                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3b5cf79000-7f3b5cf7a000 rw-p 00016000 08:01 2752516                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f3b5cf7a000-7f3b5d07d000 r-xp 00000000 08:01 2752524                    /lib/x86_64-linux-gnu/libm-2.24.so
7f3b5d07d000-7f3b5d27c000 ---p 00103000 08:01 2752524                    /lib/x86_64-linux-gnu/libm-2.24.so
7f3b5d27c000-7f3b5d27d000 r--p 00102000 08:01 2752524                    /lib/x86_64-linux-gnu/libm-2.24.so
7f3b5d27d000-7f3b5d27e000 rw-p 00103000 08:01 2752524                    /lib/x86_64-linux-gnu/libm-2.24.so
7f3b5d27e000-7f3b5d3f0000 r-xp 00000000 08:01 10229266                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f3b5d3f0000-7f3b5d5f0000 ---p 00172000 08:01 10229266                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f3b5d5f0000-7f3b5d5fa000 r--p 00172000 08:01 10229266                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f3b5d5fa000-7f3b5d5fc000 rw-p 0017c000 08:01 10229266                   /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f3b5d5fc000-7f3b5d600000 rw-p 00000000 00:00 0
7f3b5d600000-7f3b5d626000 r-xp 00000000 08:01 2752604                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f3b5d626000-7f3b5d825000 ---p 00026000 08:01 2752604                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f3b5d825000-7f3b5d829000 r--p 00025000 08:01 2752604                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f3b5d829000-7f3b5d82a000 rw-p 00029000 08:01 2752604                    /lib/x86_64-linux-gnu/libtinfo.so.5.9
7f3b5d82a000-7f3b5d85b000 r-xp 00000000 08:01 10230780                   /usr/lib/x86_64-linux-gnu/libedit.so.2.0.55
7f3b5d85b000-7f3b5da5b000 ---p 00031000 08:01 10230780                   /usr/lib/x86_64-linux-gnu/libedit.so.2.0.55
7f3b5da5b000-7f3b5da5d000 r--p 00031000 08:01 10230780                   /usr/lib/x86_64-linux-gnu/libedit.so.2.0.55
7f3b5da5d000-7f3b5da5e000 rw-p 00033000 08:01 10230780                   /usr/lib/x86_64-linux-gnu/libedit.so.2.0.55
7f3b5da5e000-7f3b5da62000 rw-p 00000000 00:00 0
7f3b5da62000-7f3b5da6b000 r-xp 00000000 08:01 10233109                   /usr/lib/x86_64-linux-gnu/libyajl.so.2.1.0
7f3b5da6b000-7f3b5dc6a000 ---p 00009000 08:01 10233109                   /usr/lib/x86_64-linux-gnu/libyajl.so.2.1.0
7f3b5dc6a000-7f3b5dc6b000 r--p 00008000 08:01 10233109                   /usr/lib/x86_64-linux-gnu/libyajl.so.2.1.0
7f3b5dc6b000-7f3b5dc6c000 rw-p 00009000 08:01 10233109                   /usr/lib/x86_64-linux-gnu/libyajl.so.2.1.0
7f3b5dc6c000-7f3b5ded1000 r-xp 00000000 08:01 10229375                   /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f3b5ded1000-7f3b5e0d0000 ---p 00265000 08:01 10229375                   /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f3b5e0d0000-7f3b5e0ee000 r--p 00264000 08:01 10229375                   /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f3b5e0ee000-7f3b5e0fc000 rw-p 00282000 08:01 10229375                   /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f3b5e0fc000-7f3b5e0ff000 rw-p 00000000 00:00 0
7f3b5e0ff000-7f3b5e161000 r-xp 00000000 08:01 10229376                   /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f3b5e161000-7f3b5e361000 ---p 00062000 08:01 10229376                   /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f3b5e361000-7f3b5e365000 r--p 00062000 08:01 10229376                   /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f3b5e365000-7f3b5e36b000 rw-p 00066000 08:01 10229376                   /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f3b5e36b000-7f3b5e383000 r-xp 00000000 08:01 2752535                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7f3b5e383000-7f3b5e582000 ---p 00018000 08:01 2752535                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7f3b5e582000-7f3b5e583000 r--p 00017000 08:01 2752535                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7f3b5e583000-7f3b5e584000 rw-p 00018000 08:01 2752535                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7f3b5e584000-7f3b5e588000 rw-p 00000000 00:00 0
7f3b5e588000-7f3b5e589000 r-xp 00000000 08:01 10232429                   /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.62.0
7f3b5e589000-7f3b5e788000 ---p 00001000 08:01 10232429                   /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.62.0
7f3b5e788000-7f3b5e789000 r--p 00000000 08:01 10232429                   /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.62.0
7f3b5e789000-7f3b5e78a000 rw-p 00001000 08:01 10232429                   /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.62.0
7f3b5e78a000-7f3b5e799000 r-xp 00000000 08:01 10232625                   /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.62.0
7f3b5e799000-7f3b5e999000 ---p 0000f000 08:01 10232625                   /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.62.0
7f3b5e999000-7f3b5e99a000 r--p 0000f000 08:01 10232625                   /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.62.0
7f3b5e99a000-7f3b5e99b000 rw-p 00010000 08:01 10232625                   /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.62.0
7f3b5e99b000-7f3b5e9a0000 r-xp 00000000 08:01 10232433                   /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0
7f3b5e9a0000-7f3b5eba0000 ---p 00005000 08:01 10232433                   /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0
7f3b5eba0000-7f3b5eba1000 r--p 00005000 08:01 10232433                   /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0
7f3b5eba1000-7f3b5eba2000 rw-p 00006000 08:01 10232433                   /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0
7f3b5eba2000-7f3b5ecb4000 r-xp 00000000 08:01 10233097                   /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0
7f3b5ecb4000-7f3b5eeb3000 ---p 00112000 08:01 10233097                   /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0
7f3b5eeb3000-7f3b5eeb8000 r--p 00111000 08:01 10233097                   /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0
7f3b5eeb8000-7f3b5eeba000 rw-p 00116000 08:01 10233097                   /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0
7f3b5eeba000-7f3b5eebb000 rw-p 00000000 00:00 0
7f3b5eebb000-7f3b5ef36000 r-xp 00000000 08:01 10232627                   /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0
7f3b5ef36000-7f3b5f136000 ---p 0007b000 08:01 10232627                   /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0
7f3b5f136000-7f3b5f139000 r--p 0007b000 08:01 10232627                   /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0
7f3b5f139000-7f3b5f13a000 rw-p 0007e000 08:01 10232627                   /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0
7f3b5f13a000-7f3b5f13d000 r-xp 00000000 08:01 10232431                   /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0
7f3b5f13d000-7f3b5f33c000 ---p 00003000 08:01 10232431                   /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0
7f3b5f33c000-7f3b5f33d000 r--p 00002000 08:01 10232431                   /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0
7f3b5f33d000-7f3b5f33e000 rw-p 00003000 08:01 10232431                   /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0
7f3b5f33e000-7f3b5f364000 r-xp 00000000 08:01 10233099                   /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
7f3b5f364000-7f3b5f563000 ---p 00026000 08:01 10233099                   /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
7f3b5f563000-7f3b5f565000 r--p 00025000 08:01 10233099                   /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
7f3b5f565000-7f3b5f566000 rw-p 00027000 08:01 10233099                   /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
7f3b5f566000-7f3b5f569000 r-xp 00000000 08:01 2752523                    /lib/x86_64-linux-gnu/libdl-2.24.so
7f3b5f569000-7f3b5f768000 ---p 00003000 08:01 2752523                    /lib/x86_64-linux-gnu/libdl-2.24.so
7f3b5f768000-7f3b5f769000 r--p 00002000 08:01 2752523                    /lib/x86_64-linux-gnu/libdl-2.24.so
7f3b5f769000-7f3b5f76a000 rw-p 00003000 08:01 2752523                    /lib/x86_64-linux-gnu/libdl-2.24.so
7f3b5f76a000-7f3b5f78d000 r-xp 00000000 08:01 2752515                    /lib/x86_64-linux-gnu/ld-2.24.so
7f3b5f79b000-7f3b5f79c000 ---p 00000000 00:00 0
7f3b5f79c000-7f3b5f7dc000 rw-p 00000000 00:00 0
7f3b5f7dc000-7f3b5f7dd000 ---p 00000000 00:00 0
7f3b5f7dd000-7f3b5f81d000 rw-p 00000000 00:00 0
7f3b5f81d000-7f3b5f81e000 ---p 00000000 00:00 0
7f3b5f81e000-7f3b5f85e000 rw-p 00000000 00:00 0
7f3b5f85e000-7f3b5f85f000 ---p 00000000 00:00 0
7f3b5f85f000-7f3b5f89f000 rw-p 00000000 00:00 0
7f3b5f89f000-7f3b5f8a0000 ---p 00000000 00:00 0
7f3b5f8a0000-7f3b5f8f4000 rw-p 00000000 00:00 0
7f3b5f8f4000-7f3b5f978000 r-xp 00000000 08:01 2753196                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
7f3b5f978000-7f3b5f979000 ---p 00084000 08:01 2753196                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
7f3b5f979000-7f3b5f97c000 r--p 00084000 08:01 2753196                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
7f3b5f97c000-7f3b5f97d000 rw-p 00087000 08:01 2753196                    /lib/x86_64-linux-gnu/libsystemd.so.0.17.0
7f3b5f97d000-7f3b5f97e000 rw-p 00000000 00:00 0
7f3b5f97e000-7f3b5f982000 rw-p 00000000 00:00 0
7f3b5f989000-7f3b5f98d000 rw-p 00000000 00:00 0
7f3b5f98d000-7f3b5f98e000 r--p 00023000 08:01 2752515                    /lib/x86_64-linux-gnu/ld-2.24.so
7f3b5f98e000-7f3b5f98f000 rw-p 00024000 08:01 2752515                    /lib/x86_64-linux-gnu/ld-2.24.so
7f3b5f98f000-7f3b5f990000 rw-p 00000000 00:00 0
7ffefe43b000-7ffefe45c000 rw-p 00000000 00:00 0                          [stack]
7ffefe5d6000-7ffefe5d8000 r--p 00000000 00:00 0                          [vvar]
7ffefe5d8000-7ffefe5da000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Caught SIGABRT.
Current time: 2018-12-02 06:32:10 +0100

/var/log/icinga2/icinga2.log.1

[2018-12-02 06:28:39 +0100] information/WorkQueue: #16 (JsonRpcConnection, #7) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-12-02 06:28:40 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:29:00 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:29:30 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:29:40 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:29:59 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-12-02 06:30:00 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:30:23 +0100] information/WorkQueue: #13 (JsonRpcConnection, #4) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-12-02 06:30:28 +0100] information/WorkQueue: #11 (JsonRpcConnection, #2) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-12-02 06:30:30 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-12-02 06:30:30 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:30:40 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:31:00 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:31:15 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2018-12-02 06:31:30 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:31:40 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-12-02 06:31:40 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:32:00 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-12-02 06:32:02 +0100] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0.15/s (9/min 45/5min 134/15min);

Let me know if you need more things.

Napsty commented 5 years ago

Just experienced (probably) the same crash on a fresh install (only two weeks old). Due to its semi-production state, I didn't see that Icinga2 wasn't running anymore for the last 2 days. Only after a login into icingaweb2 the UI informed me that the backend is not running.

There was a report file created in /var/log/icinga2/crash:

# ls -la /var/log/icinga2/crash
total 24
drwxr-s--x 2 nagios adm 4096 Dec 22 06:25 .
drwxr-s--x 4 nagios adm 4096 Dec 23 06:25 ..
-rw-r--r-- 1 nagios adm 1529 Dec  8 13:06 report.1544270800.969342
-rw-r--r-- 1 nagios adm 1529 Dec  8 13:06 report.1544270814.937787
-rw-r--r-- 1 nagios adm 1529 Dec  8 13:07 report.1544270853.915306
-rw-r--r-- 1 nagios adm 2524 Dec 22 06:25 report.1545456308.298044

Content:

# cat /var/log/icinga2/crash/report.1545456308.298044 
  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.18.0-0.bpo.1-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 486e413fb159

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x14cae1342fff]
    (1) libc.so.6: abort (+0x16a) [0x14cae134442a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x14cae1380c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x14cae1386fc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x14cae138780e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa4) [0x5575de6591b4]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x5575de60e45a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x5575de5eb8f2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x5575de7f4b64]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x5575de5e98a1]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x5575de7c4c19]
    (11) icinga2: <unknown function> (+0x266446) [0x5575de5ef446]
    (12) icinga2: main (+0x49) [0x5575de593889]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x14cae13302e1]
    (14) icinga2: _start (+0x2a) [0x5575de5b0d6a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory
NielsH commented 5 years ago

While not a pretty fix, this does help us sleep at night so perhaps it will help someone. We added the following file:

# /lib/systemd/system/icinga2.service.d/00-restart.conf
[Service]
Restart=on-failure

This will automatically restart Icinga2 while it crashes. Make sure to run systemctl daemon-reload after placing the file.

Napsty commented 5 years ago

@NielsH Thanks for sharing this. Please note that user-adjusted Systemd configs should reside in /etc/systemd, not /lib/systemd. So I placed your workaround here: /etc/systemd/system/icinga2.service.d/00-restart.conf

jandd commented 5 years ago

I have the same issue on Debian 8 and 9 systemd machines/VMs, with 2.10.2-1. The icinga2 instances on a few machines crash once in a while. Thanks @NielsH @Napsty I'll apply your workaround until the root cause of the issue is identified/fixed.

Elias481 commented 5 years ago

We also encouter such issues with logrotate with icinga2 agents updated recently. At least the 2.9.0 seems to be fine but the 2.10 version we have around show the problem. We have the issue on Centos 7 with systemd and Centos 6 without systemd. We can see the crash happens diretly after logrotate created the new empty log file and probably sent SIGUSR1 immediately before crash.

From one Centos 6 system I now got a core dump after fixing abrtd config (but for some reason again no crash report from icinga2). I doubt that for this case (similiar to at least one of the above) the double-free fix will help but assume as it looks the crash happens during creation of new timer object. And I can see that the time module was heavily changed between 2.9.0 and 2.9.1 regarding initialiszation/threading etc. so it might be worth to take a look on that part.


#0  icinga::Timer::__base_dtor (this=0x31d6f60) at /usr/include/boost153/boost/signals2/detail/slot_groups.hpp:95
#1  0x0000000000c4c729 in icinga::Timer::__deleting_dtor (this=0x31d6f60) at ../base/timer.cpp:83
#2  0x0000000000c95f14 in __base_dtor  (this=0x2ac164000cb0, stream=Unhandled dwarf expression opcode 0xf3
) at /usr/include/boost153/boost/smart_ptr/intrusive_ptr.hpp:97
#3  operator= (this=0x2ac164000cb0, stream=Unhandled dwarf expression opcode 0xf3
) at /usr/include/boost153/boost/smart_ptr/intrusive_ptr.hpp:135
#4  BindStream (this=0x2ac164000cb0, stream=Unhandled dwarf expression opcode 0xf3
) at ../base/streamlogger.cpp:75
#5  0x0000000000c96385 in ReopenLogFile (this=0x2ac164000cb0) at ../base/filelogger.cpp:75
#6  0x0000000000cf7e91 in operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/function/function_template.hpp:767
#7  m_invoke (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:89
#8  operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:78
#9  m_invoke (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:114
#10 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:107
#11 dereference (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/slot_call_iterator.hpp:82
#12 dereference (this=0x2ac130000a10) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:514
#13 operator* (this=0x2ac130000a10) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:639
#14 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/optional_last_value.hpp:55
#15 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/result_type_wrapper.hpp:64
#16 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:246
#17 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:695
#18 icinga::Application::RunEventLoop (this=0x2ac130000a10) at ../base/application.cpp:320
#19 0x0000000000cf9659 in icinga::IcingaApplication::Main (this=0x2ac130000a10) at ../icinga/icingaapplication.cpp:119
#20 0x0000000000d4e303 in icinga::Application::Run (this=0x2ac130000a10) at ../base/application.cpp:1033
#21 0x0000000000e7ffc8 in Run (this=Unhandled dwarf expression opcode 0xf3
) at ../cli/daemoncommand.cpp:320
#22 0x0000000000e8c99c in Main () at icinga.cpp:681
#23 0x0000000000a34eb7 in main (argc=10, argv=0x7ffddae8f3e8) at icinga.cpp:975```
dnsmichi commented 5 years ago

Thanks for sharing your findings. I'm leaving this as feedback for others to share more insights. This is a nasty one, on my TODO list and I am working to figure out the underlaying problem.

Elias481 commented 5 years ago

We had exactly same picture on next icinga2 "logrotate"-crash on one of the machines I monitor for this issue. I now took some time to review changes to timer module but in general the workflow did not change in a way that could directly explain these now appearing errors as far as I can see.

But if I'm correct with my limit understanding of C++ and boost magic - and icinga2 code - the following can happen:

The same thing could happen with older code 2.9.0 etc. - but the thread timing and memory reuse pattern which can influence whether a hard crash happens on duplicate delete (or just some minor memory corruption, somehow gracful handled excpetion or even nothing bad) could be changed by many parts of the code. So if my understanding regarding the reference counting and magical destruction of unused object by reference count is correct this is a thing that needs to be fixed anyway as doing delete twice leads to undefined behaviour including such "random" crashes and it was just luck when it did not happen with older code.

It shouldn't be too nasty to fix that when knowing C++ and boost and icinga2 code structure. Without much knowledge about them I assume one can just omit the Line 321 in timer.cpp and do the bind in 332 directly to the plain pointer? The destructor will wait for the timer to finish anyway (first on the mutex hold by timer thread and afterwards on the m_Running field) so no need to have a instrusive_ptr for this - and without instrusive_ptr also no decrement/duplicate free not done.

Elias481 commented 5 years ago

So and now we got few (3) more coredumps over the weekend. Two of them are quite interesting as it looks like they shows exactly the issue as I assumed - but in the case the destructor from thread pool is a bit slower and the one that runs into error because abject already deleted. (The third is again the crash in main thread as above.)

#0  icinga::Timer::__base_dtor (this=0x23bcf00)
    at /usr/include/boost153/boost/signals2/detail/slot_groups.hpp:95
#1  0x0000000000c4c729 in icinga::Timer::__deleting_dtor (this=0x23bcf00)
    at ../base/timer.cpp:83
#2  0x0000000000d527a8 in __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3
)
    at /usr/include/boost153/boost/smart_ptr/intrusive_ptr.hpp:97
#3  __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3
)
    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tuple:128
#4  __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3
)
    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tuple:229
#5  __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3
)
    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tuple:388
#6  __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3
)
    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:1280
#7  std::_Function_base::_Base_manager<std::_Bind<std::_Mem_fn<void (icinga::Timer::*)()> ()(boost::intrusive_ptr<icinga::Timer>)> >::_M_destroy (__dest=Unhandled dwarf expression opcode 0xf3
)
    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:1926
#8  std::_Function_base::_Base_manager::_M_manager (__dest=Unhandled dwarf expression opcode 0xf3
)
    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:1950
#9  0x0000000000cff25c in __base_dtor (this=0x131a9a8, queue=...)
    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:2030
#10 __base_dtor (this=0x131a9a8, queue=...)
    at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:2174
#11 __base_dtor (this=0x131a9a8, queue=...) at ../base/threadpool.hpp:68
#12 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x131a9a8, queue=...)
    at ../base/threadpool.cpp:184
#13 0x00000034ee80c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
#14 0x00000034e9a07aa1 in start_thread () from /lib64/libpthread.so.0
#15 0x00000034e96e8c4d in clone () from /lib64/libc.so.6

For reference we can see in such cases the Main Thread that is exactly at that line (but not the one catching the SEGFAULT):

#0  boost::signals2::detail::signal_impl::connect (this=0x2388400, slot=..., position=boost::signals2::at_back) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:160
#1  0x0000000000c9606a in connect (this=0x2b8fc0002680, stream=Unhandled dwarf expression opcode 0xf3
) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:664
#2  BindStream (this=0x2b8fc0002680, stream=Unhandled dwarf expression opcode 0xf3
) at ../base/streamlogger.cpp:77
#3  0x0000000000c96385 in ReopenLogFile (this=0x2b8fc0002680) at ../base/filelogger.cpp:75
#4  0x0000000000cf7e91 in operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/function/function_template.hpp:767
#5  m_invoke (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:89
#6  operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:78
#7  m_invoke (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:114
#8  operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:107
#9  dereference (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/slot_call_iterator.hpp:82
#10 dereference (this=0x2b8fc0000c80) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:514
#11 operator* (this=0x2b8fc0000c80) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:639
#12 operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/optional_last_value.hpp:55
#13 operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/result_type_wrapper.hpp:64
#14 operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:246
#15 operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:695
#16 icinga::Application::RunEventLoop (this=0x2b8fc0000c80) at ../base/application.cpp:320
#17 0x0000000000cf9659 in icinga::IcingaApplication::Main (this=0x2b8fc0000c80) at ../icinga/icingaapplication.cpp:119
#18 0x0000000000d4e303 in icinga::Application::Run (this=0x2b8fc0000c80) at ../base/application.cpp:1033
#19 0x0000000000e7ffc8 in Run (this=Unhandled dwarf expression opcode 0xf3
) at ../cli/daemoncommand.cpp:320
#20 0x0000000000e8c99c in Main () at icinga.cpp:681
#21 0x0000000000a34eb7 in main (argc=10, argv=0x7ffc16e17ae8) at icinga.cpp:975
Elias481 commented 5 years ago

I found some time to verify the issue we have with icinga2 dying on SIGUSR1. If the threads are synchronized "properly" (like so: https://github.com/Elias481/icinga2/compare/master...Elias481:bugverify/logrotate-segfault-6737) it is very reproducible (each SIGUSR1 leads to the SEGFAULT).

Also I verified that these can be fixed with different approaches:

And there are probably some more ways to fix this issue..

Al2Klimov commented 5 years ago

Hello guys and thank you very much for investigation!

Please could you test the linked PR?

Best, AK

Elias481 commented 5 years ago

I doubt that this will helps for this as the issue of possible re-incrementing refcount after it dropped to zero remains persistent regardless of the change visible in that in commit. But I can merge it on my bugverify branch to verif that.

Al2Klimov commented 5 years ago

Yes, please!

Elias481 commented 5 years ago

As expected this changes nothing (the bind was not involved at all).

If You did what You wrote with Your commit (use C++11 lambda instead of shared pointer) it could have been a fix if You really removed the causing Ptr allocs as in https://github.com/Elias481/icinga2/compare/master...Elias481:fix/logrotate-segfault-6737 but You can see that no change from bind to lambda is needed to archive that.

But You only did what You told in Your PRs (Timer::TimerThreadProc(): use C++11 lambda instead of bind()) that changed nothing (Ptr's involved are still the same - or possibly one less, but if there is really one less the one that's omitted now never counted toward the problem).

If You want to verify Yourself feel free to use https://github.com/Elias481/icinga2/tree/bugverify/logrotate-segfault-6737-testproposal to do that.

Al2Klimov commented 5 years ago

All the others will be changed as well: #7006

Elias481 commented 5 years ago

But thats is not related to this issue...

Elias481 commented 5 years ago

We now had another error with possibly/hopefully same cause, this time on a satellite with of course more concurrent activity as on the agents:

*** Error in `/usr/lib64/icinga2/sbin/icinga2': malloc(): memory corruption (fast): 0x00000000029f4320 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5d4)[0x2b8d02b385d4]
/lib64/libc.so.6(+0x82d10)[0x2b8d02b3bd10]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x2b8d02b3e1ac]
/usr/lib64/icinga2/sbin/icinga2[0x5f94bf]
/usr/lib64/icinga2/sbin/icinga2[0x5f955b]
/usr/lib64/icinga2/sbin/icinga2(yajl_parse+0x49)[0x82cd79]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga10JsonDecodeERKNS_6StringE+0x3e1)[0x82d551]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga7JsonRpc13DecodeMessageERKNS_6StringE+0x23)[0x879ed3]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga17JsonRpcConnection14MessageHandlerERKNS_6StringE+0x37)[0x8acc17]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga17JsonRpcConnection21MessageHandlerWrapperERKNS_6StringE+0x5b)[0x9873db]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga9WorkQueue15RunTaskFunctionERKSt8functionIFvvEE+0x29)[0x779ff9]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga9WorkQueue16WorkerThreadProcEv+0x5e7)[0x780c57]
/lib64/libboost_thread-mt.so.1.53.0(+0xd27a)[0x2b8d0099027a]
/lib64/libpthread.so.0(+0x7dd5)[0x2b8d028a4dd5]
/lib64/libc.so.6(clone+0x6d)[0x2b8d02bb6ead]

Stacktrace:

        (0) libc.so.6: gsignal (+0x37) [0x2b8d02aef207]
        (1) libc.so.6: abort (+0x148) [0x2b8d02af08f8]
        (2) libc.so.6: <unknown function> (+0x78d27) [0x2b8d02b31d27]
        (3) libc.so.6: <unknown function> (+0x7f5d4) [0x2b8d02b385d4]
        (4) libc.so.6: <unknown function> (+0x82d10) [0x2b8d02b3bd10]
        (5) libc.so.6: __libc_malloc (+0x4c) [0x2b8d02b3e1ac]
        (6) /usr/lib64/icinga2/sbin/icinga2() [0x5f94bf]
        (7) /usr/lib64/icinga2/sbin/icinga2() [0x5f955b]
        (8) icinga2: yajl_parse (+0x49) [0x82cd79]
        (9) icinga2: icinga::JsonDecode(icinga::String const&) (+0x3e1) [0x82d551]
        (10) icinga2: icinga::JsonRpc::DecodeMessage(icinga::String const&) (+0x23) [0x879ed3]
        (11) icinga2: icinga::JsonRpcConnection::MessageHandler(icinga::String const&) (+0x37) [0x8acc17]
        (12) icinga2: icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) (+0x5b) [0x9873db]
        (13) icinga2: icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) (+0x29) [0x779ff9]
        (14) icinga2: icinga::WorkQueue::WorkerThreadProc() (+0x5e7) [0x780c57]
        (15) libboost_thread-mt.so.1.53.0: <unknown function> (+0xd27a) [0x2b8d0099027a]
        (16) libpthread.so.0: <unknown function> (+0x7dd5) [0x2b8d028a4dd5]
        (17) libc.so.6: clone (+0x6d) [0x2b8d02bb6ead]

Sadly no coredump or debug info available (core dumps not configured for suid processes and processes that cannot write to cwd and gdb not isntalled).

I assume in this case the same happended as in the other cases but before the double-free with 2nd delete occurred another thread tried to allocate some memory of still running thread. At least I do not find another valid reason for this to happen at exactly logrotate time.

dnsmichi commented 5 years ago

I believe that this originates from YAJL leaking memory. YAJL is the JSON parser library we've been using, and now have replaced in 2.11 with #7019. You might want to give the snapshot packages a try, to see whether this problem has been solved.

https://icinga.com/docs/icinga2/snapshot/doc/21-development/#snapshot-packages-nightly-builds