JSON-RPC Crashes with 2.11

[lazyfrosch]

Task List

• [ ] Reproduce the problem
  • [x] in progress Bisect the commits and deploy each of them into a Terraform/Ansible cloud environment and let them run for 1-3 days until it crashes
  • [ ] Analyse the crashing commit and create possible patches
  • [ ] Test patches
• [ ] Test mitigations - snapshot packages: https://icinga.com/docs/icinga2/latest/doc/21-development/#snapshot-packages-nightly-builds

Mitigations

https://github.com/Icinga/icinga2/issues/7532#issuecomment-565510038

• [x] Update nlohmann/json #7712
• [x] Increase coroutine stack size #7713
• [x] Revert NotificationResult #7737

Analysis

• Problem: https://github.com/Icinga/icinga2/issues/7532#issuecomment-562626020
• Update: https://github.com/Icinga/icinga2/issues/7532#issuecomment-564000336
• Memory Analysis: https://github.com/Icinga/icinga2/issues/7532#issuecomment-564023459 & https://github.com/Icinga/icinga2/issues/7532#issuecomment-564119881

Related issues

7569

7687

7624

7470

ref/NC/636691 ref/NC/644339 ref/NC/644553 ref/NC/647127 ref/NC/652035 ref/NC/652071 ref/NC/652087

Original Report

The setup is a dual master system which was upgraded to 2.11 around noon yesterday.

In the late evening crashes started to appear and are now consistent. The system ran on 2.11-rc1 before.

The user started upgrading agents to 2.11, this may be related.

ref/NC/636691

Latest crash

  Application version: r2.11.0-1

System information:
  Platform: Ubuntu
  Platform version: 18.04.3 LTS (Bionic Beaver)
  Kernel: Linux
  Kernel version: 4.15.0-1050-aws
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: runner-LTrJQZ9N-project-298-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

        (0) libc.so.6: gsignal (+0xc7) [0x7f9a99cb4e97]
        (1) libc.so.6: abort (+0x141) [0x7f9a99cb6801]
        (2) libc.so.6: <unknown function> (+0x3039a) [0x7f9a99ca639a]
        (3) libc.so.6: <unknown function> (+0x30412) [0x7f9a99ca6412]
        (4) icinga2: <unknown function> (+0x3656c3) [0x55c11ec5d6c3]
        (5) icinga2: icinga::NotificationComponent::NotificationTimerHandler() (+0x1116) [0x55c11ec77a06]
        (6) icinga2: <unknown function> (+0x6da759) [0x55c11efd2759]
        (7) icinga2: icinga::Timer::Call() (+0x2d) [0x55c11eff151d]
        (8) icinga2: <unknown function> (+0x6f0acd) [0x55c11efe8acd]
        (9) icinga2: boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<icinga::ThreadPool::Post<std::function<void ()> >(std::function<void ()>, icinga::SchedulerPolicy)::{lambda()#1}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, std::allocator<void>*, boost::system::error_code const&, unsigned long) (+0xc5) [0x55c11ef9e5d5]
        (10) icinga2: <unknown function> (+0x75807b) [0x55c11f05007b]
        (11) icinga2: <unknown function> (+0x6a2555) [0x55c11ef9a555]
        (12) icinga2: boost_asio_detail_posix_thread_function (+0xf) [0x55c11f047f3f]
        (13) libpthread.so.0: <unknown function> (+0x76db) [0x7f9a98a756db]
        (14) libc.so.6: clone (+0x3f) [0x7f9a99d9788f]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
The program is not being run.

Alternative crashes

        (0) libc.so.6: gsignal (+0xc7) [0x7f32355c0e97]
        (1) libc.so.6: abort (+0x141) [0x7f32355c2801]
        (2) libc.so.6: <unknown function> (+0x89897) [0x7f323560b897]
        (3) libc.so.6: <unknown function> (+0x9090a) [0x7f323561290a]
        (4) libc.so.6: cfree (+0x4dc) [0x7f3235619e2c]
        (5) icinga2: <unknown function> (+0x6d90d1) [0x56119dd120d1]
        (6) icinga2: icinga::JsonRpcConnection::SendMessageInternal(boost::intrusive_ptr<icinga::Dictionary> const&) (+0x4f) [0x56119dc4870f]
        (7) icinga2: <unknown function> (+0x5f6eac) [0x56119dc2feac]
        (8) icinga2: boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (+0x75) [0x56119dc8ce85]
        (9) icinga2: <unknown function> (+0x75807b) [0x56119dd9107b]
        (10) icinga2: icinga::IoEngine::RunEventLoop() (+0x5e) [0x56119dd85b1e]
        (11) libstdc++.so.6: <unknown function> (+0xbd66f) [0x7f323349b66f]
        (12) libpthread.so.0: <unknown function> (+0x76db) [0x7f32343816db]
        (13) libc.so.6: clone (+0x3f) [0x7f32356a388f]

[lazyfrosch]

I was not able to run Icinga 2 on GCC or attach to it:

/build/gdb-JPMZNV/gdb-8.1/gdb/dictionary.c:690: internal-error: void insert_symbol_hashed(dictionary*, symbol*): Assertion `SYMBOL_LANGUAGE (sym) == DICT_LANGUAGE (dict)->la_language' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.

[lazyfrosch]

After disabling the notification feature and some minutes running:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000563ab115b5de in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_local_data (
    this=<optimized out>, this=<optimized out>) at /usr/include/c++/8/ext/new_allocator.h:81
81      /usr/include/c++/8/ext/new_allocator.h: No such file or directory.
[Current thread is 1 (Thread 0x7f72580d4700 (LWP 20323))]
(gdb) bt
#0  0x0000563ab115b5de in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_local_data (
    this=<optimized out>, this=<optimized out>) at /usr/include/c++/8/ext/new_allocator.h:81
#1  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string (__str=...,
    this=<optimized out>, this=<optimized out>, __str=...) at /usr/include/c++/8/bits/basic_string.h:542
#2  icinga::String::String (other=..., this=<optimized out>, this=<optimized out>, other=...) at ./lib/base/string.cpp:37
#3  __gnu_cxx::new_allocator<icinga::String>::construct<icinga::String, icinga::String> (this=0x7f7220590c88,
    __p=0x7473756c43273a27) at /usr/include/c++/8/ext/new_allocator.h:136
#4  std::allocator_traits<std::allocator<icinga::String> >::construct<icinga::String, icinga::String> (__a=...,
    __p=0x7473756c43273a27) at /usr/include/c++/8/bits/alloc_traits.h:475
#5  std::vector<icinga::String, std::allocator<icinga::String> >::emplace_back<icinga::String> (this=<optimized out>)
    at /usr/include/c++/8/bits/vector.tcc:103
#6  0x0000563ab108d70f in icinga::JsonRpcConnection::SendMessageInternal(boost::intrusive_ptr<icinga::Dictionary> const&) ()
    at ./lib/remote/jsonrpcconnection.cpp:186
#7  0x0000563ab1074eac in boost::asio::detail::completion_handler<icinga::JsonRpcConnection::SendMessage(boost::intrusive_ptr<icinga::Dictionary> const&)::{lambda()#1}>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) [clone .lto_priv.4649] () at ./lib/remote/jsonrpcconnection.cpp:173
#8  0x0000563ab10d1e85 in boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) () at /usr/include/icinga-boost/boost/asio/detail/scheduler_operation.hpp:40
/build/gdb-JPMZNV/gdb-8.1/gdb/dictionary.c:690: internal-error: void insert_symbol_hashed(dictionary*, symbol*): Assertion `SYMBOL_LANGUAGE (sym) == DICT_LANGUAGE (dict)->la_language' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.

[bunghi]

I have the exactly same issue. Today i've upgraded from 2.10 to 2.11 and the service started to crash. I also noticed that used memory is way higher than before. I'm thinking to downgrade..

[lazyfrosch]

@bunghi please share logs and details, "exactly" the same won't help us...

[bunghi]

The Icinga2 environment looks like this:

• web server running icingaweb2
• dual node master
• 13 zones (one endpoint for all but one with 2 endpoints)

This morning we upgraded icinga2 on all 15 servers (masters + zones endpoints). Since then the both endpoints in the zone with 2 crashed (there we have a lot of hosts).

Crash reports:

$ cat report.1569317444.555398
Caught unhandled exception.
Current time: 2019-09-24 11:30:44 +0200

  Application version: r2.11.0-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-11-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: runner-LTrJQZ9N-project-298-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Error: Function call 'opendir' for file '/var/lib/icinga2/api/zones-stage//global' failed with error code 2, 'No such file or directory'

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Failed to launch GDB: No such file or directory

$ cat report.1569312889.121968
Caught unhandled exception.
Current time: 2019-09-24 10:14:49 +0200

  Application version: r2.11.0-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-11-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: runner-LTrJQZ9N-project-298-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Error: [json.exception.parse_error.101] parse error at line 1, column 28: syntax error while parsing value - unexpected '{'; expected end of input

        (0) icinga2: icinga::JsonDecode(icinga::String const&) (+0x779) [0x55e373ddfac9]
        (1) icinga2: icinga::Process::DoEvents() (+0x295) [0x55e373e33bf5]
        (2) icinga2: icinga::Process::IOThreadProc(int) (+0x3cd) [0x55e373e3529d]
        (3) libstdc++.so.6: <unknown function> (+0xb9e6f) [0x7fa157ae2e6f]
        (4) libpthread.so.0: <unknown function> (+0x74a4) [0x7fa1589194a4]
        (5) libc.so.6: clone (+0x3f) [0x7fa157257d0f]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Failed to launch GDB: No such file or directory

Since upgraded this morning it crashed 5 times with Out of Memory kernel error. kern_node1.log kern_node2.log

Memory usage after upgrade:

[bunghi]

I don't know if I should open a different issue for performance degradation after upgrade. Both RAM and CPU usage increased:

[dnsmichi]

Memory and CPU usage are expected to raise with the introduction of user land threads with Boost Coroutines. This is different to this issue.

Also, the Json error between main and spawn helper process is a new issue, please move this into a dedicated issue, as requested in https://github.com/Icinga/icinga2/issues/7531#issuecomment-534547311 The error with opendir also is different, new issue please.

[Al2Klimov]

@dnsmichi It's not because of the dependencies, otherwise this config would crash:

for (i in range(259)) {
    object Host i {
        check_command = "dummy"
        check_interval = 5s
    }

    object Dependency i use (i) {
        parent_host_name = i
        child_host_name = (i + 1) % 259
        disable_checks = true
    }
}

[dnsmichi]

It could be related to JsonEncode seen in various other places as well, maybe related to how memory is allocated and later free'd for encoding dictionaries.

[bunghi]

Hi,

Today it crashed again, after a while.. maybe output helps:

$ cat report.1571908186.877839
  Application version: r2.11.1-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-11-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: runner-LTrJQZ9N-project-298-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

        (0) libc.so.6: gsignal (+0xcf) [0x7f1a2f41dfff]
        (1) libc.so.6: abort (+0x16a) [0x7f1a2f41f42a]
        (2) libc.so.6: <unknown function> (+0x70c00) [0x7f1a2f45bc00]
        (3) libc.so.6: <unknown function> (+0x76fc6) [0x7f1a2f461fc6]
        (4) libc.so.6: <unknown function> (+0x7780e) [0x7f1a2f46280e]
        (5) icinga2: icinga::ObjectImpl<icinga::CheckResult>::~ObjectImpl() (+0x7f) [0x558be62d3e1f]
        (6) icinga2: <unknown function> (+0x55b123) [0x558be6363123]
        (7) icinga2: icinga::Checkable::ProcessCheckResult(boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::MessageOrigin> const&) (+0x17a2) [0x558be62f9502]
        (8) icinga2: icinga::ClusterEvents::CheckResultAPIHandler(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) (+0xa6c) [0x558be62fbb7c]
        (9) icinga2: std::_Function_handler<icinga::Value (boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&), icinga::Value (*)(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&)>::_M_invoke(std::_Any_data const&, boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) (+0x23) [0x558be6205713]
        (10) icinga2: icinga::JsonRpcConnection::MessageHandler(icinga::String const&) (+0x531) [0x558be6151451]
        (11) icinga2: icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0x283) [0x558be615ad03]
        (12) icinga2: <unknown function> (+0x43a18d) [0x558be624218d]
        (13) libboost_context.so.1.67.0: make_fcontext (+0x2f) [0x7f1a31dad72f]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

[dnsmichi]

@lippserd @Al2Klimov my suspicion is that this is related to the JSON library with encode/decode, likewise object serialization and a possible leak in there. I haven't run Valgrind yet, but this would be the next thing to try.

[Baboon92]

ref/NC/644339

[mwaldmueller]

ref/NC/644553

[MarkNReynolds]

I am experiencing exactly the same crash output as @bunghi posted above. I have a 2 node master cluster and around 200 agent instances. Prior to the upgrade to 2.11 the masters were stable, now both nodes crash several times a day.

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-LTrJQZ9N-project-322-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

        (0) libc.so.6: gsignal (+0x37) [0x7f3a87108337]
        (1) libc.so.6: abort (+0x148) [0x7f3a87109a28]
        (2) libc.so.6: <unknown function> (+0x78e87) [0x7f3a8714ae87]
        (3) libc.so.6: <unknown function> (+0x7f7c4) [0x7f3a871517c4]
        (4) libc.so.6: <unknown function> (+0x82f00) [0x7f3a87154f00]
        (5) libc.so.6: __libc_malloc (+0x4c) [0x7f3a87157adc]
        (6) libstdc++.so.6: operator new(unsigned long) (+0x1d) [0x7f3a87c32ecd]
        (7) /usr/lib64/icinga2/sbin/icinga2() [0x66093f]
        (8) /usr/lib64/icinga2/sbin/icinga2() [0x6b9c12]
        (9) icinga2: icinga::JsonDecode(icinga::String const&) (+0x4ad) [0x917b2d]
        (10) icinga2: icinga::ApiListener::ReplayLog(boost::intrusive_ptr<icinga::JsonRpcConnection> const&) (+0xa4d) [0xb4497d]
        (11) icinga2: icinga::ApiListener::SyncClient(boost::intrusive_ptr<icinga::JsonRpcConnection> const&, boost::intrusive_ptr<icinga::Endpoint> const&, bool) (+0x61f) [0xb4699f]
        (12) /usr/lib64/icinga2/sbin/icinga2() [0xb47a7a]
        (13) libboost_context.so.1.69.0: make_fcontext (+0x2f) [0x7f3a89bf718f]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

I've installed gdb on the masters to see if it provides any useful details.

Mark

[carraroj]

ref/NC/647127

[Napsty]

Yesterday (Nov 14th 2019) we experienced the same crash as @bunghi mentioned. Dual master setup here, 4 zones.

Both masters run 2.11.2-1.xenial.

Caught unhandled exception.
Current time: 2019-11-14 15:30:05 +0100

  Application version: r2.11.2-1

System information:
  Platform: Ubuntu
  Platform version: 16.04.6 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.4.0-101-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.4.0
  Build host: runner-LTrJQZ9N-project-298-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Error: [json.exception.parse_error.101] parse error at line 1, column 27: syntax error while parsing value - unexpected '{'; expected end of input

        (0) icinga2: icinga::JsonDecode(icinga::String const&) (+0xcb2) [0x5fca02]
        (1) icinga2: icinga::Process::DoEvents() (+0x27c) [0x63b85c]
        (2) icinga2: icinga::Process::IOThreadProc(int) (+0x3b7) [0x6400a7]
        (3) libstdc++.so.6: <unknown function> (+0xb8c80) [0x7fb42e48ec80]
        (4) libpthread.so.0: <unknown function> (+0x76ba) [0x7fb42f0456ba]
        (5) libc.so.6: clone (+0x6d) [0x7fb43035041d]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Failed to launch GDB: No such file or directory

[lippserd]

Hi all,

We are able to reproduce the crash and are working on a fix since quite some time. Most of the time was/is used on gathering insights why this is happening. There is no estimate for a fix unfortunately. Though we are working hard on resolving this as soon as possible. Thanks to @dnsmichi for the following summary. Please read on:

Json-RPC Crash Analysis

Author: @dnsmichi Started: 2019-09-25 Last Updated: 2019-12-06

All GitHub issue and customer ticket stack traces and insights. First seen within a HA cluster with notifications and pki requests.

Observation: Increased memory usage with 2.11. Observation: Boost ASIO and JSON related.

Origins

JsonRPC Messages

IoEngine

RunEventLoop switches between coroutines, JsonRpcConnection::SendMessageInternal is called and a cfree() crashes. Leads to memory corruption.

(0) libc.so.6: gsignal (+0xc7) [0x7f32355c0e97]
       (1) libc.so.6: abort (+0x141) [0x7f32355c2801]
       (2) libc.so.6: <unknown function> (+0x89897) [0x7f323560b897]
       (3) libc.so.6: <unknown function> (+0x9090a) [0x7f323561290a]
       (4) libc.so.6: cfree (+0x4dc) [0x7f3235619e2c]
       (5) icinga2: <unknown function> (+0x6d90d1) [0x56119dd120d1]
       (6) icinga2: icinga::JsonRpcConnection::SendMessageInternal(boost::intrusive_ptr<icinga::Dictionary> const&) (+0x4f) [0x56119dc4870f]
       (7) icinga2: <unknown function> (+0x5f6eac) [0x56119dc2feac]
       (8) icinga2: boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (+0x75) [0x56119dc8ce85]
       (9) icinga2: <unknown function> (+0x75807b) [0x56119dd9107b]
       (10) icinga2: icinga::IoEngine::RunEventLoop() (+0x5e) [0x56119dd85b1e]
       (11) libstdc++.so.6: <unknown function> (+0xbd66f) [0x7f323349b66f]
       (12) libpthread.so.0: <unknown function> (+0x76db) [0x7f32343816db]
       (13) libc.so.6: clone (+0x3f) [0x7f32356a388f]

• https://github.com/Icinga/icinga2/issues/7532

Heartbeats

JsonRpcConnection::HandleAndWriteHeartbeats

calls JsonRpcConnection::SendMessageInternal.

(0) libc.so.6: gsignal (+0xc7) [0x7f1c47a35e97]
(1) libc.so.6: abort (+0x141) [0x7f1c47a37801]
(2) libc.so.6: <unknown function> (+0x3039a) [0x7f1c47a2739a]
(3) libc.so.6: <unknown function> (+0x30412) [0x7f1c47a27412]
(4) icinga2: <unknown function> (+0x578603) [0x55aadd980603]
(5) icinga2: icinga::JsonRpcConnection::HandleAndWriteHeartbeats(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0x960) [0x55aadda20270]
(6) icinga2: <unknown function> (+0x5e277d) [0x55aadd9ea77d]
(7) libboost_context.so.1.67.0: make_fcontext (+0x2f) [0x7f1c4509a65f]

• https://github.com/Icinga/icinga2/issues/7569

Thread Pool

ASIO based implementation.

Calls Timer which crashes inside the NotificationTimerHandler (this invokes the reminder notifications).

(0) libc.so.6: gsignal (+0xc7) [0x7f9a99cb4e97]
(1) libc.so.6: abort (+0x141) [0x7f9a99cb6801]
(2) libc.so.6: <unknown function> (+0x3039a) [0x7f9a99ca639a]
(3) libc.so.6: <unknown function> (+0x30412) [0x7f9a99ca6412]
(4) icinga2: <unknown function> (+0x3656c3) [0x55c11ec5d6c3]
(5) icinga2: icinga::NotificationComponent::NotificationTimerHandler() (+0x1116) [0x55c11ec77a06]
(6) icinga2: <unknown function> (+0x6da759) [0x55c11efd2759]
(7) icinga2: icinga::Timer::Call() (+0x2d) [0x55c11eff151d]
(8) icinga2: <unknown function> (+0x6f0acd) [0x55c11efe8acd]
(9) icinga2: boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<icinga::ThreadPool::Post<std::function<void ()> >(std::function<void ()>, icinga::SchedulerPolicy)::{lambda()#1}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, std::allocator<void>*, boost::system::error_code const&, unsigned long) (+0xc5) [0x55c11ef9e5d5]
(10) icinga2: <unknown function> (+0x75807b) [0x55c11f05007b]
(11) icinga2: <unknown function> (+0x6a2555) [0x55c11ef9a555]
(12) icinga2: boost_asio_detail_posix_thread_function (+0xf) [0x55c11f047f3f]
(13) libpthread.so.0: <unknown function> (+0x76db) [0x7f9a98a756db]
(14) libc.so.6: clone (+0x3f) [0x7f9a99d9788f]

• https://github.com/Icinga/icinga2/issues/7532

JsonDecode

Points to char buffer corruption.

Error: [json.exception.parse_error.101] parse error at line 1, column 28: syntax error while parsing value - unexpected '{'; expected end of input

        (0) icinga2: icinga::JsonDecode(icinga::String const&) (+0x779) [0x55e373ddfac9]

https://github.com/Icinga/icinga2/issues/7532#issuecomment-534542773

Points to corruption with large file chunks read from JSON.

(0) libc.so.6: gsignal (+0x37) [0x7f3a87108337]
(1) libc.so.6: abort (+0x148) [0x7f3a87109a28]
(2) libc.so.6: <unknown function> (+0x78e87) [0x7f3a8714ae87]
(3) libc.so.6: <unknown function> (+0x7f7c4) [0x7f3a871517c4]
(4) libc.so.6: <unknown function> (+0x82f00) [0x7f3a87154f00]
(5) libc.so.6: __libc_malloc (+0x4c) [0x7f3a87157adc]
(6) libstdc++.so.6: operator new(unsigned long) (+0x1d) [0x7f3a87c32ecd]
(7) /usr/lib64/icinga2/sbin/icinga2() [0x66093f]
(8) /usr/lib64/icinga2/sbin/icinga2() [0x6b9c12]
(9) icinga2: icinga::JsonDecode(icinga::String const&) (+0x4ad) [0x917b2d]
(10) icinga2: icinga::ApiListener::ReplayLog(boost::intrusive_ptr<icinga::JsonRpcConnection> const&) (+0xa4d) [0xb4497d]
(11) icinga2: icinga::ApiListener::SyncClient(boost::intrusive_ptr<icinga::JsonRpcConnection> const&, boost::intrusive_ptr<icinga::Endpoint> const&, bool) (+0x61f) [0xb4699f]

https://github.com/Icinga/icinga2/issues/7532#issuecomment-549272700

PKI Requests which save a JSON file

Involves JsonEncode().

[2019-10-25 08:06:25 +0200] warning/JsonRpcConnection: Error while processing message for identity 'agent42.keksi.com'
Error: basic_ios::clear: iostream error

        (0) libstdc++.so.6: <unknown function> (+0xfbfca) [0x7ff5fd73ffca]
        (1) icinga2: icinga::Utility::SaveJsonFile(icinga::String const&, int, icinga::Value const&) (+0x198) [0x96d3f8]
        (2) /usr/lib64/icinga2/sbin/icinga2() [0xa87347]
        (3) icinga2: std::_Function_handler<icinga::Value (boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&), icinga::Value (*)(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&)>::_M_invoke(std::_Any_data const&, boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) (+0xf) [0xabe06f]
        (4) icinga2: icinga::JsonRpcConnection::MessageHandler(icinga::String const&) (+0x490) [0xa8da00]
        (5) icinga2: icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0x10c) [0xab2b6c]
        (6) /usr/lib64/icinga2/sbin/icinga2() [0xab48b1]
        (7) libboost_context.so.1.69.0: make_fcontext (+0x2f) [0x7ff5ff7343af]

ref/NC/644339

CheckResult handling in the Cluster

(0) libc.so.6: gsignal (+0xcf) [0x7f1a2f41dfff]
(1) libc.so.6: abort (+0x16a) [0x7f1a2f41f42a]
(2) libc.so.6: <unknown function> (+0x70c00) [0x7f1a2f45bc00]
(3) libc.so.6: <unknown function> (+0x76fc6) [0x7f1a2f461fc6]
(4) libc.so.6: <unknown function> (+0x7780e) [0x7f1a2f46280e]
(5) icinga2: icinga::ObjectImpl<icinga::CheckResult>::~ObjectImpl() (+0x7f) [0x558be62d3e1f]
(6) icinga2: <unknown function> (+0x55b123) [0x558be6363123]
(7) icinga2: icinga::Checkable::ProcessCheckResult(boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::MessageOrigin> const&) (+0x17a2) [0x558be62f9502]
(8) icinga2: icinga::ClusterEvents::CheckResultAPIHandler(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) (+0xa6c) [0x558be62fbb7c]
(9) icinga2: std::_Function_handler<icinga::Value (boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&), icinga::Value (*)(boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&)>::_M_invoke(std::_Any_data const&, boost::intrusive_ptr<icinga::MessageOrigin> const&, boost::intrusive_ptr<icinga::Dictionary> const&) (+0x23) [0x558be6205713]
(10) icinga2: icinga::JsonRpcConnection::MessageHandler(icinga::String const&) (+0x531) [0x558be6151451]
(11) icinga2: icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0x283) [0x558be615ad03]
(12) icinga2: <unknown function> (+0x43a18d) [0x558be624218d]

This involves Deserialize() calls which invoke JsonDecode().

• https://github.com/Icinga/icinga2/issues/7532#issuecomment-545841104

Changes with possible influence

Replaced JSON library which may corrupt the stack from serialize/deserialize

https://github.com/Icinga/icinga2/pull/7019

Json is involved in any crash above, this is priority number one.

Scenario Tasks:

• Fire many node setup pki requests against an HA cluster and/or single instance. WIP, being analysed by @dnsmichi
• Or have many (docker) clients which try to sign a certificate WIP, being analysed by @al2klimov
• Or fire large nested check results into the cluster
• Or have a lot of replay log (maybe hard to trigger)

Maybe also create a standalone PoC which uses Icinga code and just stresses the JsonEncode/JsonDecode in a multi-threaded and asio-coroutine involved scenario.

Replaced UTF8 Library

May have an influence, comes in combination with the JSON library updates.

Task: Take this into account while debugging.

https://github.com/Icinga/icinga2/pull/7014

NotificationResult being passed inside the cluster

Crashes happen even when notifications are disabled. OnNotificationRequested() is relayed throughout the cluster, low possible influence here.

Network stack in general, specifically the latest changes for Coroutine Spawn for Nessus Scan Crashes - Disconnects, etc.

Maybe, but the error pattern does not match in this regard. Then we would see direct connection crashes on every connect.

Fixes which may have an influence

• SSL Context not reloaded in memory for agents waiting for a signed CSR. This would increase the number of open sockets and maybe influence the behaviour on the master. https://github.com/Icinga/icinga2/pull/7654

Things to proof: Many open sockets cause JsonRPC to crash from within Boost Asio/Context?

• Connections kept open after requesting a certificate. https://github.com/Icinga/icinga2/pull/7686

Things to proof: None. The PR needs to be tested and merged, in combination with the other crashes.

[widhalmt]

As an idea to reproduce a huge replay log: Configure a zone with 2 nodes in a HA-setup and shut one node down for quite a while. When you restart it, there should be a significant size of replay log, especially if you send a lot of data to the remaining node.

[Al2Klimov]

Update

At the moment I'm bisecting 2.10..2.11 in the setup I've already described. This will take some time as one bisect step takes the following times:

• Terraform destroy/apply
• Ansible playbook
• Crash interval

But this bisect step time can be halved by doubling the amount of setups (we're working on this together with @mxhash). Yes, two times a whole Icinga env with 2 masters and 300 agents.

So far I've figured out that the RC1 was already affected.

[dnsmichi]

Memory Analysis Test Setup

cd centos7-dev
vagrant up
vagrant ssh

i2_debug 

chown -R icinga:icinga /usr/local/icinga2/

icinga2 api setup

cp /usr/local/icinga2/var/lib/icinga2/certs//icinga2-centos7-dev.vagrant.demo.icinga.com.crt /vagrant/

/usr/local/icinga2/lib/icinga2/prepare-dirs /usr/local/icinga2/etc/sysconfig/icinga2

yum -y install valgrind 

valgrind --trace-children=yes --leak-check=full --show-leak-kinds=all --log-file=valgrind-`date +%s`.log /usr/local/icinga2/lib64/icinga2/sbin/icinga2 daemon

Second terminal.

cd centos7-dev 
mkdir -p etc/icinga2/features-enabled
mkdir -p etc/icinga2/features-available 

while true; do icinga2 node setup -DSysconfDir=`pwd`/etc -DLocalStateDir=`pwd`/var --cn a1 --zone a1 --endpoint p1 --parent_zone master --parent_host 192.168.33.21 --trustedcert icinga2-centos7-dev.vagrant.demo.icinga.com.crt; done 

Results

• ConfigCompiler leaks are expected, that's a one shot not freeing the sandbox afterwards (memory fragmentation), with the config objects holding references to debug info
• RingBuffers store stats, one time allocation
• Boost via OpenSSL Initialize - runtime malloc
• Boost Asio Coroutines use a default stack size of 64KB, this is treated as false positive
• Thread init loses some bytes (Timer thread, Threadpool, etc.)
• LazyInit'ing the IoEngine instance is also expected

==22248== Memcheck, a memory error detector
==22248== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==22248== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==22248== Command: /usr/local/icinga2/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon
==22248== Parent PID: 16971
==22248==
==22280== Warning: client switching stacks?  SP change: 0x154ea2e0 --> 0x14cd13b0
==22280==          to suppress, use: --max-stackframe=8490800 or greater
==22280== Warning: client switching stacks?  SP change: 0x14cd0de0 --> 0x154ea2e0
==22280==          to suppress, use: --max-stackframe=8492288 or greater
==22321==
==22321== HEAP SUMMARY:
==22321==     in use at exit: 6,613,370 bytes in 119,590 blocks
==22321==   total heap usage: 739,271 allocs, 619,681 frees, 105,255,095 bytes allocated

==22316== LEAK SUMMARY:
==22316==    definitely lost: 2,480 bytes in 28 blocks
==22316==    indirectly lost: 517 bytes in 8 blocks
==22316==      possibly lost: 143,232 bytes in 50 blocks
==22316==    still reachable: 6,461,871 bytes in 119,441 blocks
==22316==                       of which reachable via heuristic:
==22316==                         stdstring          : 2,114,986 bytes in 34,598 blocks
==22316==                         newarray           : 1,648 bytes in 1 blocks
==22316==         suppressed: 0 bytes in 0 blocks
==22316==
==22316== For counts of detected and suppressed errors, rerun with: -v
==22316== ERROR SUMMARY: 34 errors from 34 contexts (suppressed: 0 from 0)

Ideas

One thing which comes to mind with large buffers, while looking into valgrind logs:

• The current coroutine stack size uses the default of 64KB on Linux/Unix, this was only raised for Windows agents after the debug session after RC1. The Linux parts never changed from the original implementation.
• On the Windows side, specific stack guards and page protection was in place, with raising the stack size this had an influence.
• For testing purposes, I'd suggest raising the stack size limit @Al2Klimov @lippserd

https://github.com/Icinga/icinga2/blob/master/lib/base/io-engine.hpp#L104

I'll continue with memory analysis with massif/heaptrack.

[Al2Klimov]

@dnsmichi I've added this to my to do list.

TODO

• [x] bisect
• [x] turn off 200/300 agents
• [x] newer JSON lib
• [ ] old JSON lib
• [ ] bad ticket
• [ ] wrong ticket
• [x] stack size

[dnsmichi]

Memory Analysis with Heaptrack

https://milianw.de/blog/heaptrack-a-heap-memory-profiler-for-linux.html

Actually a small CMake patch is needed, https://github.com/KDE/heaptrack/pull/23

yum -y install libdwarf-devel elfutils libunwind-devel libzstd-devel

cd
# git clone https://github.com/KDE/heaptrack
git clone https://github.com/dnsmichi/heaptrack 
cd heaptrack

git checkout bugfix/cmake-boost-zstd 

mkdir build
cd build

cmake -DCMAKE_BUILD_TYPE=Release -DBoost_NO_BOOST_CMAKE=TRUE -DBoost_NO_SYSTEM_PATHS=TRUE -DBOOST_LIBRARYDIR=/usr/lib64/boost169 -DBOOST_INCLUDEDIR=/usr/include/boost169 -DBoost_ADDITIONAL_VERSIONS='1.69;1.69.0' ..

make 

cd ..

./build/bin/heaptrack /usr/local/icinga2/lib64/icinga2/sbin/icinga2 daemon

Stress Test

while true; do sleep 0.1 && icinga2 node setup -DSysconfDir=`pwd`/etc -DLocalStateDir=`pwd`/var --cn a1 --zone a1 --endpoint p1 --parent_zone master --parent_host 192.168.33.21 --trustedcert icinga2-centos7-dev.vagrant.demo.icinga.com.crt &>/dev/null &  done

Results

./build/bin/heaptrack --analyze "/root/heaptrack/heaptrack.icinga2.26504.gz"

Nothing new thus far. I'll continue on Friday.

[dnsmichi]

Since I cannot reproduce the crash here, I'm going a different route with things we can try together to help mitigate the issue.

Stack Rlimit

Anyone affected by the problem, please try this first!

By default, Icinga tries to lower the stack limit. This can be superseded by adding the following parameter to the systemd service file or init script:

--no-stack-rlimit

In case of Systemd, copy the existing service file and then modify it accordingly.

cp /usr/lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service
vim /etc/systemd/system/icinga2.service

ExecStart=/usr/sbin/icinga2 daemon --no-stack-rlimit --close-stdio -e ${ICINGA2_ERROR_LOG}

systemctl daemon-reload

systemctl restart icinga2

ps aux | grep icinga2
icinga   25282  0.0  0.4 368984  8692 ?        Ssl  17:18   0:00 /usr/lib64/icinga2/sbin/icinga2 daemon --no-stack-rlimit --close-stdio -e /var/log/icinga2/error.log
icinga   25294 25.5  1.2 1136692 23636 ?       Sl   17:18   0:31 /usr/lib64/icinga2/sbin/icinga2 daemon --no-stack-rlimit --close-stdio -e /var/log/icinga2/error.log
icinga   25316  0.3  0.8 718888 15560 ?        S    17:18   0:00 /usr/lib64/icinga2/sbin/icinga2 daemon --no-stack-rlimit --close-stdio -e /var/log/icinga2/error.log

You don't need any package update for this, just try that from within Systemd.

JSON Library Update

Next idea, merged with #7712

We use nlohmann/json 3.5.0, current is 3.7.3. The diff points to the following possibilities:

https://github.com/nlohmann/json/pull/1436

For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow.

https://github.com/nlohmann/json/issues/1708 https://github.com/nlohmann/json/pull/1722

Stack size

https://github.com/nlohmann/json/issues/1693#issuecomment-516980690

Integer Overflow

https://github.com/nlohmann/json/issues/1447

UTF8, json dump out of bounds

https://github.com/nlohmann/json/issues/1445

Coroutine stack size

PR: #7713

With #6559 I have learned that the default stack size for coroutines may to low causing crashes on Windows in this regard. The default of 64KB may be too low, especially keeping in mind that for example nlohmann/json puts an overhead of 1 pointer for each JSON object. If many of them are used within a coroutine, e.g. an agent/satellite replaying the log, the theory says that 64KB may be too low.

Our application default would be 256KB the lowest, if the application is not started with --no-stack-rlimit (which is the default param used on Debian and RHEL).

int Application::GetDefaultRLimitStack()
{
        return 256 * 1024;
}

Therefore I consider setting a Coroutine stack size of 256KB (maybe 512KB) a better default than the Boost ASIO default. This may increase the memory foot print of course, but prevent any stack corruption which likely happens here.

References I had in my pocket:

• https://github.com/boostorg/beast/issues/1055
• https://stackoverflow.com/questions/41030285/boostproperty-treeread-xml-segfaults-in-an-asio-handler-spawned-using-boost

Test Cases

• systemd no-limit-rstack
• Wait for snapshot builds from the triggered CI builds and then deploy them to your systems. https://git.icinga.com/packaging/rpm-icinga2/pipelines/7889 https://git.icinga.com/packaging/deb-icinga2/pipelines/7890

Instructions for testing the snapshot packages are here: https://icinga.com/docs/icinga2/latest/doc/21-development/#snapshot-packages-nightly-builds

[Al2Klimov]

For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow.

Good catch, this fits the symptoms. I'll add this to my TODO list.

[dnsmichi]

I've updated the original report with the latest findings.

[remys89]

@dnsmichi We had the same issue since this morning, after a reboot Icinga can't start anymore running via systemctl. It crashes after approx. 8/9 minutes with a lot of pidkills.

What is advised to do here? We are running version: r2.11.2-1 and are having issues with 1 of our master services now.

[lippserd]

@remys89 It would be great if you could test our snapshot packages.

[remys89]

@lippserd That would be great if you could assist in what to-do when stuff breaks even more on that server. I will discuss in the meanwhile with my team what we want to do, since it is production.

[Al2Klimov]

I've deployed a setup with not 302, but 246 machines with v2.11 + @dnsmichi's update of our JSON lib. I'll give you an update tomorrow.

[dnsmichi]

@lippserd That would be great if you could assist in what to-do when stuff breaks even more on that server.

I'm not sure if that's related to this issue, therefore I'd suggest to create a new issue including more details guided with the issue template.

Also, please enable core dumps and add the crash reports and logs to the issue.

[remys89]

@lippserd That would be great if you could assist in what to-do when stuff breaks even more on that server.

I'm not sure if that's related to this issue, therefore I'd suggest to create a new issue including more details guided with the issue template.

Also, please enable core dumps and add the crash reports and logs to the issue.

We are first restoring the server to the last working version from last friday. We configure it completely with Ansible so it should be ok. When it is up, we will configure it with ansible again and see if the effect remains.

[Al2Klimov]

The environment I've deployed yesterday has been up and running for 19h. This may or may not be enough for you as a proof that #7712 fixes #7532 and you may or may not release the RC1.

For me it's not enough as the environment significantly differs form my original one (long story). I'll deploy a definitively affected version on the same machines and give you an update tomorrow.

CC @dnsmichi @lippserd @gethash

[Al2Klimov]

Unfortunately the expected crash didn't happen tonight, so I actually didn't successfully verify #7712. I'll extend the env to the original size and give you an update tomorrow.

[dnsmichi]

Meanwhile everyone subscribed to this issue, or having the issue, please deploy the snapshot packages and verify they are working. The changeset between 2.12 and 2.11.2 is small, it only contains the IcingaDB and smaller DSL enhancements, no breaking changes at all.

[remys89]

Meanwhile everyone subscribed to this issue, or having the issue, please deploy the snapshot packages and verify they are working. The changeset between 2.12 and 2.11.2 is small, it only contains the IcingaDB and smaller DSL enhancements, no breaking changes at all.

To give you some feedback:

We noticed a lot of checks running every 5 minutes. Everytime the checks started, the CPU usage spiked hugely, crashed the Icinga2 process. We changed some intervals and went from 4 vCPU to 8 vCPU (F size in Azure).

For now, the service hasn't crashed anymore and is around 20% CPU usage. After christmas i will try this snapshot to verify if it solves the issues with a lower resourced VM.

[Al2Klimov]

In one of my two environments the master crashed as expected. This proved the environment. From now both envs will prove #7712. I'll give you an update tomorrow.

[Al2Klimov]

v2.11 + #7712 crashed. Red light for both v2.11.x and v2.12-rc1. I'll continue bisecting on the proved envs next year and give you and update ASAP.

Merry Xmas and a happy NY, AK

@gethash @lippserd @dnsmichi

[Al2Klimov]

https://github.com/Al2Klimov/icinga-iliketrains

[dnsmichi]

@Al2Klimov @lippserd #7713 with the increased coroutine stack size was not included in the tests, right?

Especially the mention of nested XML tree structures in combination with Boost ASIO sound familiar with our problem and JSON structures.

https://stackoverflow.com/questions/41030285/boostproperty-treeread-xml-segfaults-in-an-asio-handler-spawned-using-boost

If 256KB are not enough, please continue with your immediate tests and raise this limit to

• 512 KB
• 1024KB
• 2048KB
• 4096KB
• 8192KB

to mitigate whether this has an impact. On Windows, it definitely had.

https://github.com/Icinga/icinga2/pull/7713/files#diff-79f8d5c1848645a80a0fd35e6cd24c0c

[lippserd]

I tested the last current snapshots which included the PR in question. --no-stack-rlimit was active. My test setup included two masters and two satellites having the api, check, log and notification features enabled. I noticed that the config master produced a massive amount of state files. The crashes are reproducible and look like the following:

GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2...Reading symbols from /usr/lib/debug/.build-id/01/267f040293ca31203a53bba04029d07430e3c0.debug...done.
done.
[New LWP 3269]
[New LWP 3271]
[New LWP 3277]
[New LWP 3273]
[New LWP 3270]
[New LWP 3266]
[New LWP 3315]
[New LWP 3318]
[New LWP 3278]
[New LWP 3268]
[New LWP 3330]
[New LWP 3332]
[New LWP 3328]
[New LWP 3326]
[New LWP 3319]
[New LWP 3325]
[New LWP 3265]
[New LWP 3316]
[New LWP 3331]
[New LWP 3321]
[New LWP 3317]
[New LWP 3322]
[New LWP 3324]
[New LWP 3272]
[New LWP 3267]
[New LWP 3323]
[New LWP 3275]
[New LWP 3276]
[New LWP 3333]
[New LWP 3327]
[New LWP 3314]
[New LWP 3320]
[New LWP 3274]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  std::__atomic_base<unsigned long>::fetch_add (__m=std::memory_order_seq_cst, __i=1, this=0x41d77ec376281a09) at /usr/include/c++/8/bits/atomic_base.h:512
512     /usr/include/c++/8/bits/atomic_base.h: No such file or directory.
[Current thread is 1 (Thread 0x7f1a81784700 (LWP 3269))]
(gdb) bt full
#0  std::__atomic_base<unsigned long>::fetch_add (__m=std::memory_order_seq_cst, __i=1, this=0x41d77ec376281a09) at /usr/include/c++/8/bits/atomic_base.h:512
No locals.
#1  icinga::intrusive_ptr_add_ref (object=0x41d77ec376281a01) at ./lib/base/object.cpp:248
No locals.
#2  boost::intrusive_ptr<icinga::Object>::intrusive_ptr (add_ref=<optimized out>, p=<optimized out>, this=<optimized out>, this=<optimized out>, p=<optimized out>, add_ref=<optimized out>) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:69
No locals.
#3  boost::dynamic_pointer_cast<icinga::Object, icinga::Object> (p=...) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:312
No locals.
#4  _ZNK6icinga5ValuecvN5boost13intrusive_ptrIT_EEINS_6ObjectEEEv (this=<optimized out>) at ./lib/base/value.hpp:103
        __PRETTY_FUNCTION__ = <optimized out>
        object = <optimized out>
        tobject = {px = 0x41d77ec376281a01}
#5  0x0000563bd299bb2d in SerializeStack::Push (this=0x7f1a81783160, name=..., val=...) at ./lib/base/serializer.cpp:44
        obj = {px = 0x0}
        __PRETTY_FUNCTION__ = "void SerializeStack::Push(const icinga::String&, const icinga::Value&)"
#6  0x0000563bd29ac0dd in SerializeObject (stack=..., attributeTypes=4, input=...) at /usr/include/c++/8/bits/allocator.h:139
        field = {ID = 0, TypeName = 0x563bd2bb8af9 "Value", Name = 0x563bd2ac065d "command", NavigationName = 0x563bd2ac065d "command", RefTypeName = 0x0, Attributes = 4, ArrayRank = 0}
        value = {m_Value = {which_ = 4, storage_ = {<boost::detail::aligned_storage::aligned_storage_imp<32, 8>> = {data_ = {buf = "\001\032(v\303~\327A\027\000\000\000\000\000\000\000\027\000\000\000\000\000\000\000\002\000\000\000\000\000\000", align_ = {<No data fields>}}}, static size = <optimized out>, 
              static alignment = <optimized out>}}}
        i = 0
        type = {px = 0x563bd4302eb0}
        fields = std::vector of length 0, capacity 8
        olock = {m_Object = 0x7f1a280deb10, m_Locked = true}
        type = <optimized out>
        fields = <optimized out>
        olock = <optimized out>
        i = <optimized out>
        field = <optimized out>
        value = <optimized out>
#7  SerializeInternal (value=..., attributeTypes=4, stack=...) at ./lib/base/serializer.cpp:253
        input = {px = 0x7f1a280deb10}
        array = {px = 0x0}
        dict = {px = 0x0}
        ns = {px = 0x0}
#8  0x0000563bd29ac719 in icinga::Serialize (value=..., attributeTypes=<optimized out>) at ./lib/base/serializer.cpp:259

        stack = {Entries = std::deque with 0 elements}
#9  0x0000563bd279b51e in icinga::ClusterEvents::NotificationSentUserHandler (notification=..., checkable=..., user=..., notificationType=<optimized out>, cr=..., nr=..., author=..., commentText=..., command=..., origin=...) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:96
        listener = {px = 0x7f1a74004bc0}
        host = {px = 0x7f1a74363570}
        service = {px = 0x0}
        params = {px = 0x7f19fc104ba0}
        message = {px = 0x7f1a6d37b2c0}
#10 0x0000563bd2701bef in boost::function10<void, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>::operator() (a9=..., a8=..., a7=..., a6=..., a5=..., a4=..., a3=<optimized out>, a2=..., a1=..., a0=..., this=<optimized out>)
    at /usr/include/boost/function/function_template.hpp:682
No locals.
#11 boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)>, 0u, 1u, 2u, 3u, 4u, 5u, 6u, 7u, 8u, 9u, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>(boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)>&, boost::signals2::detail::unsigned_meta_array<0u, 1u, 2u, 3u, 4u, 5u, 6u, 7u, 8u, 9u>, std::tuple<boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&> const&, boost::enable_if<boost::is_void<boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)>::result_type>, void>::type*) const (this=<optimized out>, args=std::tuple containing = {...}, func=...)
    at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:105
No locals.
#12 boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::operator()<boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)>, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&, 10ul>(boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)>&, std::tuple<boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&> const&, mpl_::size_t<10ul>) const (this=<optimized out>, args=std::tuple containing = {...}, func=...)
    at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:90
No locals.
#13 boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<i--Type <RET> for more, q to quit, c to continue without paging--
nt> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > const&) const (connectionBody=..., this=0x7f1a81783620) at /usr/include/boost/signals2/detail/variadic_slot_invoker.hpp:133
No locals.
#14 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> >::dereference() const (this=0x7f1a81783420) at /usr/include/boost/signals2/detail/slot_call_iterator.hpp:110
No locals.
#15 boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > const&) (f=...) at /usr/include/boost/iterator/iterator_facade.hpp:550
No locals.
#16 boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::void_type, boost::iterators::single_pass_traversal_tag, boost::signals2::detail::void_type const&, long, false, false>::operator*() const (this=0x7f1a81783420)
    at /usr/include/boost/iterator/iterator_facade.hpp:656
No locals.
#17 boost::signals2::optional_last_value<void>::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> --Type <RET> for more, q to quit, c to continue without paging--
const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> >) const (this=<optimized out>, last=..., first=...)
    at /usr/include/boost/signals2/optional_last_value.hpp:57
No locals.
#18 boost::signals2::detail::combiner_invoker<void>::operator()<boost::signals2::optional_last_value<void>, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >(boost::signals2::optional_last_value<void>&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<boost::signals2::detail::void_type, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)> >, boost::signals2::mutex> >) const (this=<optimized out>, last=..., first=..., combiner=...) at /usr/include/boost/signals2/detail/result_type_wrapper.hpp:64
No locals.
#19 boost::signals2::detail::signal_impl<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&), boost::signals2::optional_last_value<void>, int, std::less<int>, boost::function<void (boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)>, boost::function<void (boost::signals2::connection const&, boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&)>, boost::signals2::mutex>::operator()(boost::intrusive_ptr<icinga::Notification> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::User> const&, icinga::NotificationType const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::NotificationResult> const&, icinga::String const&, icinga::String const&, icinga::String const&, boost::intrusive_ptr<icinga::MessageOrigin> const&) (this=<optimized out>, args#0=..., 
    args#1=..., args#2=..., args#3=<optimized out>, args#4=..., args#5=..., args#6=..., args#7=..., args#8=..., args#9=...) at /usr/include/boost/signals2/detail/signal_template.hpp:243
        local_state = {px = 0x563bd42ce700, pn = {pi_ = 0x563bd42ce7f0}}
        it = non-dereferenceable iterator for std::list
        invoker = {_args = std::tuple containing = {[1] = <optimized out>, [2] = <optimized out>, [3] = <optimized out>, [4] = <optimized out>, [5] = <optimized out>, [6] = <optimized out>, [7] = <optimized out>, [8] = <optimized out>, [9] = @0x7f1a81783850, [10] = @0x7f1a81783778}}
        cache = {result = {<boost::optional_detail::optional_base<boost::signals2::detail::void_type>> = {<boost::optional_detail::optional_tag> = {<No data fields>}, m_initialized = false, m_storage = {dummy_ = {data = "\035", aligner_ = 29 '\035'}}}, <No data fields>}, 
          tracked_ptrs = {<std::allocator<boost::variant<boost::shared_ptr<void>, boost::signals2::detail::foreign_void_shared_ptr> >> = {<__gnu_cxx::new_allocator<boost::variant<boost::shared_ptr<void>, boost::signals2::detail::foreign_void_shared_ptr> >> = {<No data fields>}, <No data fields>}, 
            static is_stack_buffer_empty = <optimized out>, members_ = {<boost::aligned_storage<240, 8>> = {<boost::detail::aligned_storage::aligned_storage_imp<240, 8>> = {data_ = {
                    buf = "\000NӃ\032\177\000\000\004\000\000\000\032\177\000\000\300\262\067m\032\177\000\000\n\000\000\000\000\000\000\000\220\064x\201\032\177\000\000\000\000\000\000\000\000\000\000\004\000\000\000;V\000\000\000\071\000l\032\177\000\000\350\260\060\324;V\000\000\330\065x\201\032\177\000\000\340\065x\201\032\177\000\000\004\000\000\000\032\177\000\000\220I\357\064\032\177\000\000\350\260\060\324;V\000\000)E\231\322;V\000\000\270\260\060\324;V\000\000\004\000\000\000\032\177\000\000\020\353\r(\032\177\000\000\360\204\321\322;V\000\000\204\203%v\303~\327A\000\000\000\000\032\177\000\000\001\000\000\000\031\177\000\000\000\000\000\000\000\000@@\036\000\000\000\000\000\000\000p^\324\322;V\000\000\200"..., align_ = {<No data fields>}}}, static size = <optimized out>, static alignment = <optimized out>}, capacity_ = 10}, buffer_ = 0x7f1a81783518, size_ = 0}, f = {_args = std::tuple containing = {
              [1] = @0x7f1a81783788, [2] = @0x7f1a81783780, [3] = @0x7f1a3ca91da0, [4] = @0x7f1a8178375c, [5] = @0x7f1a3ca91d98, [6] = @0x7f1a81783770, [7] = @0x7f1a3ca91d70, [8] = @0x7f1a3ca91d50, [9] = @0x7f1a81783850, [10] = @0x7f1a81783778}}, connected_slot_count = 1, disconnected_slot_count = 0, 
          m_active_slot = 0x563bd43ba640}
        janitor = {<boost::noncopyable_::noncopyable> = {<No data fields>}, _cache = @0x7f1a81783510, _sig = @0x563bd42ce6d0, _connection_bodies = 0x563bd42ce730}
        list_lock = <optimized out>
#20 0x0000563bd273fa2a in icinga::Notification::ExecuteNotificationHelper (this=<optimized out>, type=<optimized out>, user=..., cr=..., force=<optimized out>, author=..., text=...) at /usr/include/boost/smart_ptr/shared_ptr.hpp:726
        notificationName = {static NPos = 18446744073709551615, m_Data = "dummy-5508!dummy"}
        userName = {static NPos = 18446744073709551615, m_Data = "user5"}
        checkableName = {static NPos = 18446744073709551615, m_Data = "dummy-5508"}
        command = {px = 0x7f1a70000cd0}
        commandName = {static NPos = 18446744073709551615, m_Data = "dummy-notification"}
        nr = <optimized out>
        ex = <optimized out>
#21 0x0000563bd26de231 in std::__invoke_impl<void, void (icinga::Notification::*&)(icinga::NotificationType, boost::intrusive_ptr<icinga::User> const&, boost::intrusive_ptr<icinga::CheckResult> const&, bool, icinga::String const&, icinga::String const&), icinga::Notification*&, icinga::NotificationType&, boost::intrusive_ptr<icinga::User>&, boost::intrusive_ptr<icinga::CheckResult>&, bool&, icinga::String&, icinga::String&> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/8/bits/invoke.h:89
No locals.
--Type <RET> for more, q to quit, c to continue without paging--
#22 std::__invoke<void (icinga::Notification::*&)(icinga::NotificationType, boost::intrusive_ptr<icinga::User> const&, boost::intrusive_ptr<icinga::CheckResult> const&, bool, icinga::String const&, icinga::String const&), icinga::Notification*&, icinga::NotificationType&, boost::intrusive_ptr<icinga::User>&, boost::intrusive_ptr<icinga::CheckResult>&, bool&, icinga::String&, icinga::String&> (__fn=<optimized out>, __fn=<optimized out>) at /usr/include/c++/8/bits/invoke.h:95
No locals.
#23 std::_Bind<void (icinga::Notification::*(icinga::Notification*, icinga::NotificationType, boost::intrusive_ptr<icinga::User>, boost::intrusive_ptr<icinga::CheckResult>, bool, icinga::String, icinga::String))(icinga::NotificationType, boost::intrusive_ptr<icinga::User> const&, boost::intrusive_ptr<icinga::CheckResult> const&, bool, icinga::String const&, icinga::String const&)>::__call<void, , 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>) (__args=..., this=<optimized out>) at /usr/include/c++/8/functional:400
No locals.
#24 std::_Bind<void (icinga::Notification::*(icinga::Notification*, icinga::NotificationType, boost::intrusive_ptr<icinga::User>, boost::intrusive_ptr<icinga::CheckResult>, bool, icinga::String, icinga::String))(icinga::NotificationType, boost::intrusive_ptr<icinga::User> const&, boost::intrusive_ptr<icinga::CheckResult> const&, bool, icinga::String const&, icinga::String const&)>::operator()<, void>() (this=<optimized out>) at /usr/include/c++/8/functional:484
No locals.
#25 std::_Function_handler<void (), std::_Bind<void (icinga::Notification::*(icinga::Notification*, icinga::NotificationType, boost::intrusive_ptr<icinga::User>, boost::intrusive_ptr<icinga::CheckResult>, bool, icinga::String, icinga::String))(icinga::NotificationType, boost::intrusive_ptr<icinga::User> const&, boost::intrusive_ptr<icinga::CheckResult> const&, bool, icinga::String const&, icinga::String const&)> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297
No locals.
#26 0x0000563bd296aedd in icinga::ThreadPool::Post<std::function<void ()> >(std::function<void ()>, icinga::SchedulerPolicy)::{lambda()#1}::operator()() const () at /usr/include/c++/8/bits/std_function.h:682
        l_TimerMutex = <optimized out>
        l_StopTimerThread = false
        l_False = "false"
        l_True = "true"
        icinga::l_RandomMutex = <optimized out>
        l_Utf8Replacement = "�"
        icinga::l_Mutexes = <optimized out>
        l_Null = "null"
        l_TimerCV = <optimized out>
        l_TimerThread = <optimized out>
        icinga::l_SSLInitialized = true
        l_Timers = <optimized out>
        l_AliveTimers = 13
        boost::optional_ns::in_place_init_if = <optimized out>
        std::__ioinit = <optimized out>
        boost::asio::error::misc_category = <optimized out>
        boost::optional_ns::in_place_init = <optimized out>
        boost::none = <optimized out>
        boost::asio::error::addrinfo_category = <optimized out>
        boost::asio::error::netdb_category = <optimized out>
        boost::asio::error::system_category = <optimized out>
        l_StopTimerThread = <optimized out>
        l_AliveTimers = <optimized out>
        nlohmann::(anonymous namespace)::from_json = <optimized out>
        l_LastExceptionContext = <optimized out>
        l_ProcessOnceFlag = <optimized out>
        l_FDs = <optimized out>
        l_EventFDs = <optimized out>
        icinga::l_RandomMutex = <optimized out>
        icinga::l_SSLInitialized = <optimized out>
        l_LastExceptionStack = <optimized out>
        l_ProcessControlPID = <optimized out>
        l_SpawnHelperOnceFlag = <optimized out>
        l_TimerThread = <optimized out>
        (anonymous namespace)::io0::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io1::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io2::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io3::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io4::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io5::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io6::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io7::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io8::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io9::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io10::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io11::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io12::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io13::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io14::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io15::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io16::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io17::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io18::l_InitializeOnce = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        (anonymous namespace)::io19::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io20::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io21::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io22::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io23::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io24::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io25::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io26::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io27::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io28::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io29::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io30::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io31::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io32::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io33::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io34::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io35::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io36::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io37::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io38::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io39::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io40::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io41::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io42::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io43::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io44::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io45::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io46::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io47::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io48::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io49::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io50::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io51::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io52::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io53::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io54::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io55::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io56::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io57::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io58::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io59::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io60::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io61::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io62::l_InitializeOnce = <optimized out>
        (anonymous namespace)::io63::l_InitializeOnce = <optimized out>
        l_InternalNSBehavior = <optimized out>
        boost::optional_ns::in_place_init_if = <optimized out>
        boost::optional_ns::in_place_init = <optimized out>
        boost::asio::error::misc_category = <optimized out>
        boost::asio::error::system_category = <optimized out>
        std::__ioinit = <optimized out>
        boost::asio::ssl::error::stream_category = <optimized out>
        l_Restarting = <optimized out>
        l_ProcessMutex = <optimized out>
        boost::none = <optimized out>
        l_ProcessControlMutex = <optimized out>
        nlohmann::(anonymous namespace)::to_json = <optimized out>
        l_Indent = <optimized out>
        l_Timers = <optimized out>
        l_TimerMutex = <optimized out>
        l_Null = <optimized out>
        l_InExceptionHandler = <optimized out>
        boost::asio::error::ssl_category = <optimized out>
        l_TimerCV = <optimized out>
        l_EndiannessDetector = <optimized out>
        l_True = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging--
        boost::asio::error::addrinfo_category = <optimized out>
        l_Processes = <optimized out>
        l_ConsoleType = <optimized out>
        l_Utf8Replacement = <optimized out>
        l_ProcessControlFD = <optimized out>
        l_Frames = <optimized out>
        l_False = <optimized out>
        boost::asio::error::netdb_category = <optimized out>
        l_ShutdownTimersCleanlyOnExit = <optimized out>
        icinga::l_Mutexes = <optimized out>
        icinga::WorkQueue::m_NextID = {<std::__atomic_base<int>> = {static _S_alignment = 4, _M_i = 9}, <No data fields>}
        icinga::Configuration::RLimitFiles = 16384
        icinga::Configuration::AttachDebugger = false
        icinga::Configuration::RLimitStack = 262144
        icinga::Configuration::RLimitProcesses = 16384
#27 0x0000563bd292a065 in boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<icinga::ThreadPool::Post<std::function<void ()> >(std::function<void ()>, icinga::SchedulerPolicy)::{lambda()#1}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, std::allocator<void>*, boost::system::error_code const&, unsigned long) (owner=<optimized out>, base=<optimized out>) at /usr/include/boost/asio/impl/system_executor.hpp:35
        o = <optimized out>
        allocator = <optimized out>
        p = {a = <optimized out>, v = 0x0, p = 0x0}
        handler = {work_ = {executor_ = {<No data fields>}, owns_ = true}, handler_ = {__this = 0x563bd2d48740 <icinga::Application::GetTP()::tp>, __callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, 
                _M_functor = {_M_unused = {_M_object = 0x7f1a81783ca0, _M_const_object = 0x7f1a81783ca0, _M_function_pointer = 0x7f1a81783ca0, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f1a81783ca0, this adjustment 1}, 
                  _M_pod_data = "\240<x\201\032\177\000\000\001\000\000\000\000\000\000"}, _M_manager = 0x0}, _M_invoker = 0x563bd4325f80}}}
        b = <optimized out>
#28 0x0000563bd29c4f1b in boost::asio::detail::scheduler_operation::complete (bytes_transferred=<optimized out>, ec=..., owner=0x563bd4325f80, this=<optimized out>) at /usr/include/boost/asio/detail/scheduler_operation.hpp:40
No locals.
#29 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /usr/include/boost/asio/detail/impl/scheduler.ipp:401
        task_result = <optimized out>
        on_exit = <optimized out>
        o = <optimized out>
        more_handlers = true
#30 boost::asio::detail::scheduler::run (this=<optimized out>, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:154
        this_thread = {<boost::asio::detail::thread_info_base> = {<boost::asio::detail::noncopyable> = {<No data fields>}, reusable_memory_ = {0x7f1a240cc830, 0x0}}, private_op_queue = {<boost::asio::detail::noncopyable> = {<No data fields>}, front_ = 0x0, back_ = 0x0}, private_outstanding_work = 0}
        ctx = {<boost::asio::detail::noncopyable> = {<No data fields>}, key_ = 0x563bd4325f80, value_ = 0x7f1a81783d80, next_ = 0x0}
        lock = {<boost::asio::detail::noncopyable> = {<No data fields>}, mutex_ = @0x563bd4325fb0, locked_ = false}
        n = 1451553
#31 0x0000563bd28fb765 in boost::asio::thread_pool::thread_function::operator() (this=<optimized out>) at /usr/include/boost/asio/impl/thread_pool.ipp:33
        ec = {m_val = 0, m_cat = 0x7f1a843ce070}
        ec = <optimized out>
#32 boost::asio::detail::posix_thread::func<boost::asio::thread_pool::thread_function>::run (this=<optimized out>) at /usr/include/boost/asio/detail/posix_thread.hpp:86
No locals.
#33 0x0000563bd29bccff in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=<optimized out>) at /usr/include/boost/asio/detail/impl/posix_thread.ipp:74
        func = {ptr = 0x563bd4326200}
#34 0x00007f1a843d6fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
        ret = <optimized out>
        pd = <optimized out>
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139751818020608, -2113946693049274543, 140734008723230, 140734008723231, 139751818020608, 94814963045552, 2061736441401028433, 2061743542559944529}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#35 0x00007f1a839ae4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

[dnsmichi]

#9  0x0000563bd279b51e in icinga::ClusterEvents::NotificationSentUserHandler (notification=..., checkable=..., user=..., notificationType=<optimized out>, cr=..., nr=..., author=..., commentText=..., command=..., origin=...) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:96
        listener = {px = 0x7f1a74004bc0}
        host = {px = 0x7f1a74363570}
        service = {px = 0x0}
        params = {px = 0x7f19fc104ba0}
        message = {px = 0x7f1a6d37b2c0}

Since service is a nullptr, this is a host notification sync.

        notificationName = {static NPos = 18446744073709551615, m_Data = "dummy-5508!dummy"}
        userName = {static NPos = 18446744073709551615, m_Data = "user5"}
        checkableName = {static NPos = 18446744073709551615, m_Data = "dummy-5508"}
        command = {px = 0x7f1a70000cd0}
        commandName = {static NPos = 18446744073709551615, m_Data = "dummy-notification"}

@lippserd I suspect a problem with #6722 here. Since this already was part of 2.11, the analysis matches this partially. We could try reverting it and see whether this has any effect.

Keep in mind though that the original crash was not related to notifications, so my merged PRs with #7712 and #7713 may have an impact on the crash itself.

[Al2Klimov]

I've reproduced @lippserd's bug ("Notification-Result-Bug") with cd44c5371 and will bisect it independently of the JSON-RPC-Bug.

[Al2Klimov]

@lippserd I've updated https://github.com/Al2Klimov/icinga-iliketrains and my instance of it:

• node setup signs lazily to work around a problem which prevented the next bisect step up to and including today
• the nodes can open 1000000 files, just to be sure
• core dumps are enabled \o/
• packages are being cached

Now as the setup has changed I'll wait until today evening to prove it.

[Al2Klimov]

@lippserd ... and I've documented your bug:

https://github.com/Al2Klimov/icinga-ilikenotificationresults

[dnsmichi]

Does the crash still happen with git master?

[Al2Klimov]

Which crash? Mine or @lippserd's? Neither I can test mine as the only env for it is busy (for the next one or two weeks), nor the other one as the next 24 hours would cross the weekend. Shall I test @lippserd's crash monday-tuesday?

[dnsmichi]

The JSON-RPC crash from this issue. The NotificationResult crash happened during testing and was resolved with reverting the patch introduced in 2.11 (see #7737 being now merged to master).

[Al2Klimov]

Once we've done 2.11.x, we should have a look at what's wrong with that now reverted patch.

[dnsmichi]

Once we've done 2.11.x, we should have a look at what's wrong with that now reverted patch.

Since it wasn't tested in-depth nor incorporated for IcingaDB, there's no extra value or loosing value here with just leaving that as is, reverted. If IcingaDB ever needs the executed notification command line, we may re-iterate on a new design.

This doesn't answer my question though.

only env for it is busy (for the next one or two weeks)

I'd suggest to shift priorities with testing the git master then. I've discussed that possibility with @lippserd yesterday.

[Al2Klimov]

I've not even looked into the Git master due to the experience with #7712. We have < 10 bisect steps and a lot of possible shots-in-the-dark. I'm honestly working on another parallel env for the shots (which may or may not solve the problem), but I'll keep one env for bisecting (which will find the problem very likely).