dnsmichi
commented
6 years ago Can you try to reproduce this with 2.9.1? It looks fairly strange that String::Split() would cause a memory corruption here.
Closed Gleng1212 closed 5 years ago
dnsmichi
commented
6 years ago Can you try to reproduce this with 2.9.1? It looks fairly strange that String::Split() would cause a memory corruption here.
Gleng1212
commented
6 years ago I am looking to update all the servers but at this point only a few have been updated. There have a couple more failures. Only one collected crash data.
cat /var/log/icinga2/error.log-20180828 glibc detected /usr/lib64/icinga2/sbin/icinga2: malloc(): memory corruption (fast): 0x00007fe980001450 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3e1ca75dee] /lib64/libc.so.6[0x3e1ca7a4b8] /lib64/libc.so.6(__libc_malloc+0x5c)[0x3e1ca7aaac] /usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x3e286bd0cd] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga10Dictionary3SetERKNS_6StringENS_5ValueE+0xd3)[0xc06753] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga14MacroProcessor16EvaluateFunctionERKN5boost13intrusive_ptrINS_8FunctionEEERKSt6vectorISt4pairINS_6StringENS2_INS_6ObjectEEEESaISC_EERKNS2_INS_11CheckResultEEERKSt8functionIFNS_5ValueERKSM_EERKNS2_INS_10DictionaryEEEbi+0xa9)[0xd80409] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga14MacroProcessor13ResolveMacrosERKNS_5ValueERKSt6vectorISt4pairINS_6StringEN5boost13intrusive_ptrINS_6ObjectEEEESaISB_EERKNS8_INS_11CheckResultEEEPS6_RKSt8functionIFS1_S3_EERKNS8_INS_10DictionaryEEEbi+0x740)[0xd3f6f0] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga14MacroProcessor16ResolveArgumentsERKNS_5ValueERKN5boost13intrusive_ptrINS_10DictionaryEEERKSt6vectorISt4pairINS_6StringENS5_INS_6ObjectEEEESaISF_EERKNS5_INS_11CheckResultEEES9_bi+0x326)[0xd3fe16] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga13PluginUtility14ExecuteCommandERKN5boost13intrusive_ptrINS_7CommandEEERKNS2_INS_9CheckableEEERKNS2_INS_11CheckResultEEERKSt6vectorISt4pairINS_6StringENS2_INS_6ObjectEEEESaISK_EERKNS2_INS_10DictionaryEEEbRKSt8functionIFvRKNS_5ValueERKNS_13ProcessResultEEE+0xd7)[0xd41467] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga15PluginCheckTask10ScriptFuncERKN5boost13intrusive_ptrINS_9CheckableEEERKNS2_INS_11CheckResultEEERKNS2_INS_10DictionaryEEEb+0x252)[0xd421a2] /usr/lib64/icinga2/sbin/icinga2(_ZNSt17_Function_handlerIFN6icinga5ValueERKSt6vectorIS1_SaIS1_EEEZNS0_12WrapFunctionIPFvRKN5boost13intrusive_ptrINS0_9CheckableEEERKNSA_INS0_11CheckResultEEERKNSA_INS0_10DictionaryEEEbEEENSt9enable_ifIXaasrSt11is_functionINSt14remove_pointerIT_E4typeEE5valuentsrSt7is_sameISS_PS7_E5valueESt8functionIS7_EE4typeESS_EUlS6_E_E9_M_invokeERKSt9_AnydataS6+0x1a5)[0xcba505] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga8Function6InvokeERKSt6vectorINS_5ValueESaIS2_EE+0x2f)[0xbd1fcf] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga12CheckCommand7ExecuteERKN5boost13intrusive_ptrINS_9CheckableEEERKNS2_INS_11CheckResultEEERKNS2_INS_10DictionaryEEEb+0x16a)[0xbd33da] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga9Checkable12ExecuteCheckEv+0x285)[0xeb7c85] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga16CheckerComponent18ExecuteCheckHelperERKN5boost13intrusive_ptrINS_9CheckableEEE+0x1f)[0xeb8b9f] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga10ThreadPool12WorkerThread10ThreadProcERNS0_5QueueE+0x355)[0xc94d35] /usr/lib64/libboost_thread.so.1.53.0[0x3855c0c5c3] /lib64/libpthread.so.0[0x3e1ce07aa1] /lib64/libc.so.6(clone+0x6d)[0x3e1cae8bcd] ======= Memory map: ======== 00400000-010c1000 r-xp 00000000 f9:00 2503151 /usr/lib64/icinga2/sbin/icinga2 012c0000-012c3000 rw-p 00cc0000 f9:00 2503151 /usr/lib64/icinga2/sbin/icinga2 012c3000-012c9000 rw-p 00000000 00:00 0 014c2000-019a8000 rw-p 00cc2000 f9:00 2503151 /usr/lib64/icinga2/sbin/icinga2 02f59000-033c5000 rw-p 00000000 00:00 0 [heap] 033c5000-033e7000 rw-p 00000000 00:00 0 [heap] 3852600000-3852661000 r-xp 00000000 f9:00 2256471 /usr/lib64/libboost_program_options.so.1.53.0 3852661000-3852860000 ---p 00061000 f9:00 2256471 /usr/lib64/libboost_program_options.so.1.53.0 3852860000-3852865000 rw-p 00060000 f9:00 2256471 /usr/lib64/libboost_program_options.so.1.53.0 3852a00000-3852a22000 r-xp 00000000 f9:00 1703952 /lib64/libncurses.so.5.7 3852a22000-3852c21000 ---p 00022000 f9:00 1703952 /lib64/libncurses.so.5.7 3852c21000-3852c22000 rw-p 00021000 f9:00 1703952 /lib64/libncurses.so.5.7 3852e00000-3852f0e000 r-xp 00000000 f9:00 2256475 /usr/lib64/libboost_regex.so.1.53.0 3852f0e000-385310d000 ---p 0010e000 f9:00 2256475 /usr/lib64/libboost_regex.so.1.53.0 385310d000-3853114000 rw-p 0010d000 f9:00 2256475 /usr/lib64/libboost_regex.so.1.53.0 3853a00000-3853a03000 r-xp 00000000 f9:00 2256469 /usr/lib64/libboost_system.so.1.53.0 3853a03000-3853c02000 ---p 00003000 f9:00 2256469 /usr/lib64/libboost_system.so.1.53.0 3853c02000-3853c03000 rw-p 00002000 f9:00 2256469 /usr/lib64/libboost_system.so.1.53.0 3854200000-385514b000 r-xp 00000000 f9:00 2256472 /usr/lib64/libicudata.so.42.1 385514b000-385534a000 ---p 00f4b000 f9:00 2256472 /usr/lib64/libicudata.so.42.1 385534a000-385534b000 rw-p 00f4a000 f9:00 2256472 /usr/lib64/libicudata.so.42.1 3855c00000-3855c16000 r-xp 00000000 f9:00 2256470 /usr/lib64/libboost_thread.so.1.53.0 3855c16000-3855e15000 ---p 00016000 f9:00 2256470 /usr/lib64/libboost_thread.so.1.53.0 3855e15000-3855e17000 rw-p 00015000 f9:00 2256470 /usr/lib64/libboost_thread.so.1.53.0 3856000000-3856188000 r-xp 00000000 f9:00 2256474 /usr/lib64/libicui18n.so.42.1 3856188000-3856388000 ---p 00188000 f9:00 2256474 /usr/lib64/libicui18n.so.42.1 3856388000-3856395000 rw-p 00188000 f9:00 2256474 /usr/lib64/libicui18n.so.42.1 3856395000-3856396000 rw-p 00000000 00:00 0 3856400000-385653f000 r-xp 00000000 f9:00 2256473 /usr/lib64/libicuuc.so.42.1 385653f000-385673f000 ---p 0013f000 f9:00 2256473 /usr/lib64/libicuuc.so.42.1 385673f000-3856750000 rw-p 0013f000 f9:00 2256473 /usr/lib64/libicuuc.so.42.1 3856750000-3856752000 rw-p 00000000 00:00 0 3857000000-3857029000 r-xp 00000000 f9:00 2255024 /usr/lib64/libedit.so.0.0.27 3857029000-3857229000 ---p 00029000 f9:00 2255024 /usr/lib64/libedit.so.0.0.27 3857229000-385722c000 rw-p 00029000 f9:00 2255024 /usr/lib64/libedit.so.0.0.27 385722c000-385722f000 rw-p 00000000 00:00 0 3e1c200000-3e1c220000 r-xp 00000000 f9:00 1703944 /lib64/ld-2.12.so 3e1c420000-3e1c421000 r--p 00020000 f9:00 1703944 /lib64/ld-2.12.so 3e1c421000-3e1c422000 rw-p 00021000 f9:00 1703944 /lib64/ld-2.12.so 3e1c422000-3e1c423000 rw-p 00000000 00:00 0 3e1c600000-3e1c602000 r-xp 00000000 f9:00 1704175 /lib64/libdl-2.12.so 3e1c602000-3e1c802000 ---p 00002000 f9:00 1704175 /lib64/libdl-2.12.so 3e1c802000-3e1c803000 r--p 00002000 f9:00 1704175 /lib64/libdl-2.12.so 3e1c803000-3e1c804000 rw-p 00003000 f9:00 1704175 /lib64/libdl-2.12.so 3e1ca00000-3e1cb8a000 r-xp 00000000 f9:00 1704023 /lib64/libc-2.12.so 3e1cb8a000-3e1cd8a000 ---p 0018a000 f9:00 1704023 /lib64/libc-2.12.so 3e1cd8a000-3e1cd8e000 r--p 0018a000 f9:00 1704023 /lib64/libc-2.12.so 3e1cd8e000-3e1cd90000 rw-p 0018e000 f9:00 1704023 /lib64/libc-2.12.so 3e1cd90000-3e1cd94000 rw-p 00000000 00:00 0 3e1ce00000-3e1ce17000 r-xp 00000000 f9:00 1704056 /lib64/libpthread-2.12.so 3e1ce17000-3e1d017000 ---p 00017000 f9:00 1704056 /lib64/libpthread-2.12.so 3e1d017000-3e1d018000 r--p 00017000 f9:00 1704056 /lib64/libpthread-2.12.so 3e1d018000-3e1d019000 rw-p 00018000 f9:00 1704056 /lib64/libpthread-2.12.so 3e1d019000-3e1d01d000 rw-p 00000000 00:00 0 3e1d200000-3e1d207000 r-xp 00000000 f9:00 1704177 /lib64/librt-2.12.so 3e1d207000-3e1d406000 ---p 00007000 f9:00 1704177 /lib64/librt-2.12.so 3e1d406000-3e1d407000 r--p 00006000 f9:00 1704177 /lib64/librt-2.12.so 3e1d407000-3e1d408000 rw-p 00007000 f9:00 1704177 /lib64/librt-2.12.so 3e1d600000-3e1d683000 r-xp 00000000 f9:00 1704024 /lib64/libm-2.12.so 3e1d683000-3e1d882000 ---p 00083000 f9:00 1704024 /lib64/libm-2.12.so 3e1d882000-3e1d883000 r--p 00082000 f9:00 1704024 /lib64/libm-2.12.so 3e1d883000-3e1d884000 rw-p 00083000 f9:00 1704024 /lib64/libm-2.12.so 3e1da00000-3e1da15000 r-xp 00000000 f9:00 1704012 /lib64/libz.so.1.2.3 3e1da15000-3e1dc14000 ---p 00015000 f9:00 1704012 /lib64/libz.so.1.2.3 3e1dc14000-3e1dc15000 r--p 00014000 f9:00 1704012 /lib64/libz.so.1.2.3 3e1dc15000-3e1dc16000 rw-p 00015000 f9:00 1704012 /lib64/libz.so.1.2.3 3e1e200000-3e1e21d000 r-xp 00000000 f9:00 1704419 /lib64/libselinux.so.1 3e1e21d000-3e1e41c000 ---p 0001d000 f9:00 1704419 /lib64/libselinux.so.1 3e1e41c000-3e1e41d000 r--p 0001c000 f9:00 1704419 /lib64/libselinux.so.1 3e1e41d000-3e1e41e000 rw-p 0001d000 f9:00 1704419 /lib64/libselinux.so.1 3e1e41e000-3e1e41f000 rw-p 00000000 00:00 0 3e1f200000-3e1f216000 r-xp 00000000 f9:00 1704037 /lib64/libresolv-2.12.so 3e1f216000-3e1f416000 ---p 00016000 f9:00 1704037 /lib64/libresolv-2.12.so 3e1f416000-3e1f417000 r--p 00016000 f9:00 1704037 /lib64/libresolv-2.12.so 3e1f417000-3e1f418000 rw-p 00017000 f9:00 1704037 /lib64/libresolv-2.12.so 3e1f418000-3e1f41a000 rw-p 00000000 00:00 0 3e26200000-3e2621d000 r-xp 00000000 f9:00 1704452 /lib64/libtinfo.so.5.7 3e2621d000-3e2641c000 ---p 0001d000 f9:00 1704452 /lib64/libtinfo.so.5.7 3e2641c000-3e26420000 rw-p 0001c000 f9:00 1704452 /lib64/libtinfo.so.5.7 3e26420000-3e26421000 rw-p 00000000 00:00 0 3e27a00000-3e27a16000 r-xp 00000000 f9:00 1704433 /lib64/libgcc_s-4.4.7-20120601.so.1 3e27a16000-3e27c15000 ---p 00016000 f9:00 1704433 /lib64/libgcc_s-4.4.7-20120601.so.1 3e27c15000-3e27c16000 rw-p 00015000 f9:00 1704433 /lib64/libgcc_s-4.4.7-20120601.so.1 3e28600000-3e286e8000 r-xp 00000000 f9:00 2230542 /usr/lib64/libstdc++.so.6.0.13 3e286e8000-3e288e8000 ---p 000e8000 f9:00 2230542 /usr/lib64/libstdc++.so.6.0.13 3e288e8000-3e288ef000 r--p 000e8000 f9:00 2230542 /usr/lib64/libstdc++.so.6.0.13 3e288ef000-3e288f1000 rw-p 000ef000 f9:00 2230542 /usr/lib64/libstdc++.so.6.0.13 3e288f1000-3e28906000 rw-p 00000000 00:00 0 3e29a00000-3e29a03000 r-xp 00000000 f9:00 1704426 /lib64/libcom_err.so.2.1 3e29a03000-3e29c02000 ---p 00003000 f9:00 1704426 /lib64/libcom_err.so.2.1 3e29c02000-3e29c03000 r--p 00002000 f9:00 1704426 /lib64/libcom_err.so.2.1 3e29c03000-3e29c04000 rw-p 00003000 f9:00 1704426 /lib64/libcom_err.so.2.1 3e2ae00000-3e2ae02000 r-xp 00000000 f9:00 1704423 /lib64/libkeyutils.so.1.3 3e2ae02000-3e2b001000 ---p 00002000 f9:00 1704423 /lib64/libkeyutils.so.1.3 3e2b001000-3e2b002000 r--p 00001000 f9:00 1704423 /lib64/libkeyutils.so.1.3 3e2b002000-3e2b003000 rw-p 00002000 f9:00 1704423 /lib64/libkeyutils.so.1.3 3e2b600000-3e2b60a000 r-xp 00000000 f9:00 1704424 /lib64/libkrb5support.so.0.1 3e2b60a000-3e2b809000 ---p 0000a000 f9:00 1704424 /lib64/libkrb5support.so.0.1 3e2b809000-3e2b80a000 r--p 00009000 f9:00 1704424 /lib64/libkrb5support.so.0.1 3e2b80a000-3e2b80b000 rw-p 0000a000 f9:00 1704424 /lib64/libkrb5support.so.0.1 3e2ba00000-3e2bbba000 r-xp 00000000 f9:00 2231537 /usr/lib64/libcrypto.so.1.0.1e 3e2bbba000-3e2bdba000 ---p 001ba000 f9:00 2231537 /usr/lib64/libcrypto.so.1.0.1e 3e2bdba000-3e2bdd5000 r--p 001ba000 f9:00 2231537 /usr/lib64/libcrypto.so.1.0.1e 3e2bdd5000-3e2bde1000 rw-p 001d5000 f9:00 2231537 /usr/lib64/libcrypto.so.1.0.1e 3e2bde1000-3e2bde5000 rw-p 00000000 00:00 0 3e2c200000-3e2c2dc000 r-xp 00000000 f9:00 1704427 /lib64/libkrb5.so.3.3 3e2c2dc000-3e2c4db000 ---p 000dc000 f9:00 1704427 /lib64/libkrb5.so.3.3 3e2c4db000-3e2c4e5000 r--p 000db000 f9:00 1704427 /lib64/libkrb5.so.3.3 3e2c4e5000-3e2c4e7000 rw-p 000e5000 f9:00 1704427 /lib64/libkrb5.so.3.3 3e2c600000-3e2c629000 r-xp 00000000 f9:00 1704425 /lib64/libk5crypto.so.3.1 3e2c629000-3e2c829000 ---p 00029000 f9:00 1704425 /lib64/libk5crypto.so.3.1 3e2c829000-3e2c82a000 r--p 00029000 f9:00 1704425 /lib64/libk5crypto.so.3.1 3e2c82a000-3e2c82b000 rw-p 0002a000 f9:00 1704425 /lib64/libk5crypto.so.3.1 3e2c82b000-3e2c82c000 rw-p 00000000 00:00 0 3e2ca00000-3e2ca41000 r-xp 00000000 f9:00 1704428 /lib64/libgssapi_krb5.so.2.2 3e2ca41000-3e2cc41000 ---p 00041000 f9:00 1704428 /lib64/libgssapi_krb5.so.2.2 3e2cc41000-3e2cc42000 r--p 00041000 f9:00 1704428 /lib64/libgssapi_krb5.so.2.2 3e2cc42000-3e2cc44000 rw-p 00042000 f9:00 1704428 /lib64/libgssapi_krb5.so.2.2 3e2e200000-3e2e262000 r-xp 00000000 f9:00 2231539 /usr/lib64/libssl.so.1.0.1e 3e2e262000-3e2e462000 ---p 00062000 f9:00 2231539 /usr/lib64/libssl.so.1.0.1e 3e2e462000-3e2e466000 r--p 00062000 f9:00 2231539 /usr/lib64/libssl.so.1.0.1e 3e2e466000-3e2e46c000 rw-p 00066000 f9:00 2231539 /usr/lib64/libssl.so.1.0.1e 7fe918000000-7fe918021000 rw-p 00000000 00:00 0 7fe918021000-7fe91c000000 ---p 00000000 00:00 0 7fe91c000000-7fe91c021000 rw-p 00000000 00:00 0 7fe91c021000-7fe920000000 ---p 00000000 00:00 0 7fe920000000-7fe920021000 rw-p 00000000 00:00 0 7fe920021000-7fe924000000 ---p 00000000 00:00 0 7fe924000000-7fe924021000 rw-p 00000000 00:00 0 7fe924021000-7fe928000000 ---p 00000000 00:00 0 7fe928000000-7fe928021000 rw-p 00000000 00:00 0 7fe928021000-7fe92c000000 ---p 00000000 00:00 0 7fe92c000000-7fe92c021000 rw-p 00000000 00:00 0 7fe92c021000-7fe930000000 ---p 00000000 00:00 0 7fe930000000-7fe930021000 rw-p 00000000 00:00 0 7fe930021000-7fe934000000 ---p 00000000 00:00 0 7fe934000000-7fe93402a000 rw-p 00000000 00:00 0 7fe93402a000-7fe938000000 ---p 00000000 00:00 0 7fe938000000-7fe938021000 rw-p 00000000 00:00 0 7fe938021000-7fe93c000000 ---p 00000000 00:00 0 7fe93c000000-7fe93c021000 rw-p 00000000 00:00 0 7fe93c021000-7fe940000000 ---p 00000000 00:00 0 7fe940000000-7fe940021000 rw-p 00000000 00:00 0 7fe940021000-7fe944000000 ---p 00000000 00:00 0 7fe944000000-7fe944021000 rw-p 00000000 00:00 0 7fe944021000-7fe948000000 ---p 00000000 00:00 0 7fe948000000-7fe948021000 rw-p 00000000 00:00 0 7fe948021000-7fe94c000000 ---p 00000000 00:00 0 7fe94c000000-7fe94c021000 rw-p 00000000 00:00 0 7fe94c021000-7fe950000000 ---p 00000000 00:00 0 7fe950000000-7fe950021000 rw-p 00000000 00:00 0 7fe950021000-7fe954000000 ---p 00000000 00:00 0 7fe954000000-7fe954021000 rw-p 00000000 00:00 0 7fe954021000-7fe958000000 ---p 00000000 00:00 0 7fe958000000-7fe958021000 rw-p 00000000 00:00 0 7fe958021000-7fe95c000000 ---p 00000000 00:00 0 7fe95c000000-7fe95c021000 rw-p 00000000 00:00 0 7fe95c021000-7fe960000000 ---p 00000000 00:00 0 7fe960000000-7fe96005f000 rw-p 00000000 00:00 0 7fe96005f000-7fe964000000 ---p 00000000 00:00 0 7fe964000000-7fe964021000 rw-p 00000000 00:00 0 7fe964021000-7fe968000000 ---p 00000000 00:00 0 7fe968000000-7fe968065000 rw-p 00000000 00:00 0 7fe968065000-7fe968073000 rw-p 00000000 00:00 0 7fe968073000-7fe96c000000 ---p 00000000 00:00 0 7fe96c000000-7fe96c05d000 rw-p 00000000 00:00 0 7fe96c05d000-7fe970000000 ---p 00000000 00:00 0 7fe970000000-7fe970021000 rw-p 00000000 00:00 0 7fe970021000-7fe974000000 ---p 00000000 00:00 0 7fe974000000-7fe974061000 rw-p 00000000 00:00 0 7fe974061000-7fe978000000 ---p 00000000 00:00 0 7fe978000000-7fe978021000 rw-p 00000000 00:00 0 7fe978021000-7fe978025000 rw-p 00000000 00:00 0 7fe978025000-7fe97c000000 ---p 00000000 00:00 0 7fe97c000000-7fe97c021000 rw-p 00000000 00:00 0 7fe97c021000-7fe980000000 ---p 00000000 00:00 0 7fe980000000-7fe980021000 rw-p 00000000 00:00 0 7fe980021000-7fe9800a9000 rw-p 00000000 00:00 0 7fe9800a9000-7fe984000000 ---p 00000000 00:00 0 7fe984000000-7fe984021000 rw-p 00000000 00:00 0 7fe984021000-7fe984026000 rw-p 00000000 00:00 0 7fe984026000-7fe988000000 ---p 00000000 00:00 0 7fe988000000-7fe988021000 rw-p 00000000 00:00 0 7fe988021000-7fe988026000 rw-p 00000000 00:00 0 7fe988026000-7fe98c000000 ---p 00000000 00:00 0 7fe98c80a000-7fe98c80b000 ---p 00000000 00:00 0 7fe98c80b000-7fe98c84b000 rw-p 00000000 00:00 0 [stack:24276] 7fe98c84b000-7fe98c84c000 ---p 00000000 00:00 0 7fe98c84c000-7fe98c88c000 rw-p 00000000 00:00 0 [stack:24236] 7fe98c88c000-7fe98c88d000 ---p 00000000 00:00 0 7fe98c88d000-7fe98c8cd000 rw-p 00000000 00:00 0 [stack:24051] 7fe98c8cd000-7fe98c8ce000 ---p 00000000 00:00 0 7fe98c8ce000-7fe98c90e000 rw-p 00000000 00:00 0 [stack:32547] 7fe98c90e000-7fe98c90f000 ---p 00000000 00:00 0 7fe98c90f000-7fe98c94f000 rw-p 00000000 00:00 0 [stack:32546] 7fe98c94f000-7fe98c950000 ---p 00000000 00:00 0 7fe98c950000-7fe98c990000 rw-p 00000000 00:00 0 [stack:32545] 7fe98c990000-7fe98c991000 ---p 00000000 00:00 0 7fe98c991000-7fe98c9d1000 rw-p 00000000 00:00 0 [stack:32544] 7fe98c9d1000-7fe98c9d2000 ---p 00000000 00:00 0 7fe98c9d2000-7fe98ca12000 rw-p 00000000 00:00 0 [stack:32543] 7fe98ca12000-7fe98ca13000 ---p 00000000 00:00 0 7fe98ca13000-7fe98ca53000 rw-p 00000000 00:00 0 [stack:32542] 7fe98ca53000-7fe98ca54000 ---p 00000000 00:00 0 7fe98ca54000-7fe98ca94000 rw-p 00000000 00:00 0 [stack:32541] 7fe98ca94000-7fe98ca95000 ---p 00000000 00:00 0 7fe98ca95000-7fe98cad5000 rw-p 00000000 00:00 0 [stack:32540] 7fe98cad5000-7fe98cad6000 ---p 00000000 00:00 0 7fe98cad6000-7fe98cb16000 rw-p 00000000 00:00 0 [stack:32539] 7fe98cb16000-7fe98cb17000 ---p 00000000 00:00 0 7fe98cb17000-7fe98cb57000 rw-p 00000000 00:00 0 [stack:32538] 7fe98cb57000-7fe98cb58000 ---p 00000000 00:00 0 7fe98cb58000-7fe98cb98000 rw-p 00000000 00:00 0 [stack:32537] 7fe98cb98000-7fe98cb99000 ---p 00000000 00:00 0 7fe98cb99000-7fe98cbd9000 rw-p 00000000 00:00 0 [stack:32536] 7fe98cbd9000-7fe98cbda000 ---p 00000000 00:00 0 7fe98cbda000-7fe98cc1a000 rw-p 00000000 00:00 0 [stack:32535] 7fe98cc1a000-7fe98cc1b000 ---p 00000000 00:00 0 7fe98cc1b000-7fe98cc5b000 rw-p 00000000 00:00 0 7fe98cc5b000-7fe98cc5c000 ---p 00000000 00:00 0 7fe98cc5c000-7fe98cc9c000 rw-p 00000000 00:00 0 [stack:32532] 7fe98cc9c000-7fe98cc9d000 ---p 00000000 00:00 0 7fe98cc9d000-7fe98ccdd000 rw-p 00000000 00:00 0 [stack:32531] 7fe98ccdd000-7fe98ccde000 ---p 00000000 00:00 0 7fe98ccde000-7fe98cd1e000 rw-p 00000000 00:00 0 [stack:32527] 7fe98cd1e000-7fe98cd1f000 ---p 00000000 00:00 0 7fe98cd1f000-7fe98cd5f000 rw-p 00000000 00:00 0 [stack:32529] 7fe98cd5f000-7fe98cd60000 ---p 00000000 00:00 0 7fe98cd60000-7fe98cda0000 rw-p 00000000 00:00 0 [stack:32525] 7fe98cda0000-7fe98cda1000 ---p 00000000 00:00 0 7fe98cda1000-7fe98cde1000 rw-p 00000000 00:00 0 [stack:32530] 7fe98cde1000-7fe98cde2000 ---p 00000000 00:00 0 7fe98cde2000-7fe98ce22000 rw-p 00000000 00:00 0 [stack:32528] 7fe98ce22000-7fe98ce2e000 r-xp 00000000 f9:00 1703967 /lib64/libnss_files-2.12.so 7fe98ce2e000-7fe98d02d000 ---p 0000c000 f9:00 1703967 /lib64/libnss_files-2.12.so 7fe98d02d000-7fe98d02e000 r--p 0000b000 f9:00 1703967 /lib64/libnss_files-2.12.so 7fe98d02e000-7fe98d02f000 rw-p 0000c000 f9:00 1703967 /lib64/libnss_files-2.12.so 7fe98d02f000-7fe98d030000 ---p 00000000 00:00 0 7fe98d030000-7fe98d070000 rw-p 00000000 00:00 0 [stack:32519] 7fe98d070000-7fe98d071000 ---p 00000000 00:00 0 7fe98d071000-7fe98d0b1000 rw-p 00000000 00:00 0 [stack:32518] 7fe98d0b1000-7fe98d0b2000 ---p 00000000 00:00 0 7fe98d0b2000-7fe98d0f2000 rw-p 00000000 00:00 0 [stack:32517] 7fe98d0f2000-7fe98d0f3000 ---p 00000000 00:00 0 7fe98d0f3000-7fe98d133000 rw-p 00000000 00:00 0 [stack:32516] 7fe98d133000-7fe98d134000 ---p 00000000 00:00 0 7fe98d134000-7fe98d182000 rw-p 00000000 00:00 0 [stack:32520] 7fe98d18d000-7fe98d191000 rw-p 00000000 00:00 0 7fe98d191000-7fe98d193000 rw-p 00000000 00:00 0 7ffd55dc0000-7ffd55de1000 rw-p 00000000 00:00 0 [stack] 7ffd55de6000-7ffd55de8000 r--p 00000000 00:00 0 [vvar] 7ffd55de8000-7ffd55dea000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Caught SIGABRT. Current time: 2018-08-28 00:10:40 -0500
cat /var/log/icinga2/crash/report.1535433040.552201 Application information: Application version: r2.9.0-1 Installation root: /usr Sysconf directory: /etc Run directory: /var/run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /var/run/icinga2/icinga2.pid
System information: Platform: Oracle Linux Server Platform version: 6.9 Kernel: Linux Kernel version: 4.1.12-103.9.4.el6uek.x86_64 Architecture: x86_64
Build information: Compiler: GNU 4.8.2 Build host: 5009d75245b1 Stacktrace:
(0) libc.so.6: gsignal (+0x35) [0x3e1ca32495]
(1) libc.so.6: abort (+0x175) [0x3e1ca33c75]
(2) /lib64/libc.so.6() [0x3e1ca703a7]
(3) /lib64/libc.so.6() [0x3e1ca75dee]
(4) /lib64/libc.so.6() [0x3e1ca7a4b8]
(5) libc.so.6: __libc_malloc (+0x5c) [0x3e1ca7aaac]
(6) libstdc++.so.6: operator new(unsigned long) (+0x1d) [0x3e286bd0cd]
(7) icinga2: icinga::Dictionary::Set(icinga::String const&, icinga::Value) (+0xd3) [0xc06753]
(8) icinga2: icinga::MacroProcessor::EvaluateFunction(boost::intrusive_ptr<icinga::Function> const&, std::vector<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> >, std::allocator<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> > > > const&, boost::intrusive_ptr<icinga::CheckResult> const&, std::function<icinga::Value ()(icinga::Value const&)> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool, int) (+0xa9) [0xd80409]
(9) icinga2: icinga::MacroProcessor::ResolveMacros(icinga::Value const&, std::vector<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> >, std::allocator<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> > > > const&, boost::intrusive_ptr<icinga::CheckResult> const&, icinga::String*, std::function<icinga::Value ()(icinga::Value const&)> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool, int) (+0x740) [0xd3f6f0]
(10) icinga2: icinga::MacroProcessor::ResolveArguments(icinga::Value const&, boost::intrusive_ptr<icinga::Dictionary> const&, std::vector<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> >, std::allocator<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> > > > const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool, int) (+0x326) [0xd3fe16]
(11) icinga2: icinga::PluginUtility::ExecuteCommand(boost::intrusive_ptr<icinga::Command> const&, boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, std::vector<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> >, std::allocator<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> > > > const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool, std::function<void ()(icinga::Value const&, icinga::ProcessResult const&)> const&) (+0xd7) [0xd41467]
(12) icinga2: icinga::PluginCheckTask::ScriptFunc(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) (+0x252) [0xd421a2]
(13) icinga2: _ZNSt17_Function_handlerIFN6icinga5ValueERKSt6vectorIS1_SaIS1_EEEZNS0_12WrapFunctionIPFvRKN5boost13intrusive_ptrINS0_9CheckableEEERKNSA_INS0_11CheckResultEEERKNSA_INS0_10DictionaryEEEbEEENSt9enable_ifIXaasrSt11is_functionINSt14remove_pointerIT_E4typeEE5valuentsrSt7is_sameISS_PS7_E5valueESt8functionIS7_EE4typeESS_EUlS6_E_E9_M_invokeERKSt9_Any_dataS6_ (+0x1a5) [0xcba505]
(14) icinga2: icinga::Function::Invoke(std::vector<icinga::Value, std::allocator<icinga::Value> > const&) (+0x2f) [0xbd1fcf]
(15) icinga2: icinga::CheckCommand::Execute(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) (+0x16a) [0xbd33da]
(16) icinga2: icinga::Checkable::ExecuteCheck() (+0x285) [0xeb7c85]
(17) icinga2: icinga::CheckerComponent::ExecuteCheckHelper(boost::intrusive_ptr<icinga::Checkable> const&) (+0x1f) [0xeb8b9f]
(18) icinga2: icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (+0x355) [0xc94d35]
(19) /usr/lib64/libboost_thread.so.1.53.0() [0x3855c0c5c3]
(20) /lib64/libpthread.so.0() [0x3e1ce07aa1]
(21) libc.so.6: clone (+0x6d) [0x3e1cae8bcd][New LWP 24276] [New LWP 24236] [New LWP 24051] [New LWP 32547] [New LWP 32546] [New LWP 32545] [New LWP 32544] [New LWP 32543] [New LWP 32542] [New LWP 32541] [New LWP 32540] [New LWP 32539] [New LWP 32538] [New LWP 32537] [New LWP 32536] [New LWP 32535] [New LWP 32532] [New LWP 32531] [New LWP 32530] [New LWP 32529] [New LWP 32528] [New LWP 32527] [New LWP 32525] [New LWP 32520] [New LWP 32519] [New LWP 32518] [New LWP 32517] [New LWP 32516] [Thread debugging using libthread_db enabled] 0x0000003e1caacc4d in nanosleep () from /lib64/libc.so.6
Thread 29 (Thread 0x7fe98d132700 (LWP 32516)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 28 (Thread 0x7fe98d0f1700 (LWP 32517)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 27 (Thread 0x7fe98d0b0700 (LWP 32518)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 26 (Thread 0x7fe98d06f700 (LWP 32519)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 25 (Thread 0x7fe98d173700 (LWP 32520)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 24 (Thread 0x7fe98cd9f700 (LWP 32525)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 23 (Thread 0x7fe98cd1d700 (LWP 32527)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 22 (Thread 0x7fe98ce21700 (LWP 32528)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 21 (Thread 0x7fe98cd5e700 (LWP 32529)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 20 (Thread 0x7fe98cde0700 (LWP 32530)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 19 (Thread 0x7fe98ccdc700 (LWP 32531)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 18 (Thread 0x7fe98cc9b700 (LWP 32532)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 17 (Thread 0x7fe98cc19700 (LWP 32535)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 16 (Thread 0x7fe98cbd8700 (LWP 32536)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 15 (Thread 0x7fe98cb97700 (LWP 32537)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 14 (Thread 0x7fe98cb56700 (LWP 32538)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 13 (Thread 0x7fe98cb15700 (LWP 32539)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 12 (Thread 0x7fe98cad4700 (LWP 32540)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 11 (Thread 0x7fe98ca93700 (LWP 32541)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 10 (Thread 0x7fe98ca52700 (LWP 32542)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 9 (Thread 0x7fe98ca11700 (LWP 32543)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 8 (Thread 0x7fe98c9d0700 (LWP 32544)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 7 (Thread 0x7fe98c98f700 (LWP 32545)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 6 (Thread 0x7fe98c94e700 (LWP 32546)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 5 (Thread 0x7fe98c90d700 (LWP 32547)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 4 (Thread 0x7fe98c8cc700 (LWP 24051)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 3 (Thread 0x7fe98c88b700 (LWP 24236)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 2 (Thread 0x7fe98c84a700 (LWP 24276)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thread 1 (Thread 0x7fe98d1747e0 (LWP 32515)):
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
No symbol table info available.
Thanks, Glen
dnsmichi
commented
6 years ago 2.9.0 is known to have severe bugs, please test that with 2.9.1.
dnsmichi
commented
6 years ago Hm.
Platform: Oracle Linux Server
Platform version: 6.9Which package repository/source is used here?
Gleng1212
commented
6 years ago I performed the upgrade using yum update (both icinga and pcp were updated but no other packages). The previous icinga2 level was 2.8.1-1.el6.icinga.
Updating: icinga2 x86_64 2.9.0-1.el6.icinga icinga-stable-release 10 k icinga2-bin x86_64 2.9.0-1.el6.icinga icinga-stable-release 3.7 M icinga2-common x86_64 2.9.0-1.el6.icinga icinga-stable-release 149 k icinga2-libs x86_64 2.9.0-1.el6.icinga icinga-stable-release 86 k
I did update all the severs to 2.9.1-1.el6.icinga. To date there have been no failures but I have no idea what caused the crashes.
dnsmichi
commented
6 years ago And you're sure that el6 and Oracle Linux share the same system binaries, symbols and entry points? I'm inclined to say that this is the real problem here that the binaries were not built on Oracle Linux itself.
Gleng1212
commented
6 years ago As I understand it, yes they are. However I do run the UEK. Here's a link to the section of Oracle's Release Notes for Rel 6.9 listing the package differences ol_pkglists
dnsmichi
commented
6 years ago Probably you might want to try to rebuild the SRPM on Oracle and test again. Oracle Linux isn't in our supported stack, and with their proprietary model, hard to get a hold on for testing.
Gleng1212
commented
6 years ago Sorry, I'm not a developer. I believe Oracle Linux is still a free download for developers and self education. These days I think you need to create a free account and accept their "Developer Terms and Conditions". (A lot of the Oracle software is available the same way, see: Developer Downloads
I have noted that icinga is not supported on OL.
dgoetz
commented
6 years ago Problem is we build with CentOS for EPEL which in theory should be compatible with all Red Hat derivates, but from practical experience I can tell you that Oracle and Amazon sometimes use other compiler options or slightly different versions which may or may not cause problems. This is why we can not support them except from best effort as we would need to rebuild packages on them to guarantee its compatibility.
dnsmichi
commented
6 years ago Problem with all these distributions is generally: time and knowledge. I know for example that Amazon Linux also behaves differently and you need to adopt the spec file for it. With common effort from the community and developers, this may happen.
The thing we cannot do is to integrate everything into your build infrastructure, and keep that maintained on every release or snapshot build. https://www.icinga.com/support/ holds a support matrix for all distributions.
I am willing to help to make things work, and help you recompile the packages, or anything else. Though, keep in mind that such packages won't be available on packages.icinga.com without someone from the team maintaining it. A similar discussion is going on with armhf for PI, AIX, HP-UX, etc.
dnsmichi
commented
6 years ago For further problem analysis and source builds on OEL, if you want to go that route:
Al2Klimov
commented
5 years ago Hello @Gleng1212!
You don't have to be a developer to re-build source packages. That's pretty easy (if you know how to do it 😉):
First of all you have to download the RPM source package you'd like to re-build. In my case (CentOS 7):
wget 'https://packages.icinga.com/epel/7Server/release/src/icinga2-2.10.3-1.el7.icinga.src.rpm'In addition you have to install rpmbuild:
yum install rpm-buildNext try to re-build the first time:
rpmbuild --rebuild icinga2-2.10.3-1.el7.icinga.src.rpmIt will most likely complain about some missing build dependencies. Install them, too:
yum install libedit-devel ncurses-devel gcc-c++ libstdc++-devel openssl-devel bison cmake flex make boost-devel systemd-devel logrotate mysql-devel postgresql-devel checkpolicy selinux-policy-develNow re-try:
rpmbuild --rebuild icinga2-2.10.3-1.el7.icinga.src.rpmThat should purr like a cat and print some RPM file names it just created. This way you get Icinga 2 packages for the supported OSes' derivatives exactly as if we had build them ourselves.
Please try this (I'd recommend Oracle Linux 7) and the original packages and report whether custom packages fix the problem.
Best, AK
Al2Klimov
commented
5 years ago @dnsmichi Excuse me.. which "proprietary model"?
docker run -it oraclelinux:6Same for Amazon Linux:
docker run -it amazonlinux:2I consider this fixed with 2.11.
Note: /var/log/icinga2/crash/report.1534786927.216873 is included following the error.log
I have seen other client crashes but they did not produce crash reports or error.log entries. I do not know that they are relevant to this bug report but here are the /var/log/messages entries:
Aug 15 03:01:14 troubleSomeServer1 kernel: [7396627.346055] traps: icinga2[18969] trap stack segment ip:c277c7 sp:7ffdf21540d0 error:0 in icinga2 (deleted)[400000+cc1000]
Aug 15 03:05:57 troubleSomeServer2 kernel: [7395975.019082] traps: icinga2[61276] trap stack segment ip:c277c7 sp:7fff03da3770 error:0 in icinga2 (deleted)[400000+cc1000]
Aug 15 01:26:54 troubleSomeServer3 kernel: [7389949.102396] traps: icinga2[32060] general protection ip:36b9ce6295 sp:7f0fb06e2e20 error:0 in libcrypto.so.1.0.1e[36b9c00000+1ba000]
cat /var/log/icinga2/error.log glibc detected /usr/lib64/icinga2/sbin/icinga2: free(): invalid next size (fast): 0x00007fce1c0018e0 *** ======= Backtrace: ========= /lib64/libc.so.6[0x36aa875dee] /lib64/libc.so.6[0x36aa878c80] /usr/lib64/icinga2/sbin/icinga2(_ZN5boost6detail8function15functor_managerINS_9algorithm6detail13token_finderFINS410is any_ofFIcEEEEE6manageERKNS1_15function_bufferERSA_NS1_30functor_manager_operation_typeE+0x8d)[0xc1d7dd] /usr/lib64/icinga2/sbin/icinga2[0xbfdda2] /usr/lib64/icinga2/sbin/icinga2(_ZNK6icinga6String5SplitEPKc+0x1b8)[0xbfe558] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga13PluginUtility16ParseCheckOutputERKNS_6StringE+0x3a)[0xd7a28a] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga15PluginCheckTask22ProcessFinishedHandlerERKN5boost13intrusive_ptrINS_9Checkab leEEERKNS2_INS_11CheckResultEEERKNS_5ValueERKNS_13ProcessResultE+0x58)[0xebadc8] /usr/lib64/icinga2/sbin/icinga2(_ZN6icinga10ThreadPool12WorkerThread10ThreadProcERNS0_5QueueE+0x355)[0xc94d35] /usr/lib64/libboost_thread.so.1.53.0[0x3aa700c5c3] /lib64/libpthread.so.0[0x36aac07aa1] /lib64/libc.so.6(clone+0x6d)[0x36aa8e8bcd] ======= Memory map: ======== 00400000-010c1000 r-xp 00000000 f9:00 3039415 /usr/lib64/icinga2/sbin/icinga2 012c0000-012c3000 rw-p 00cc0000 f9:00 3039415 /usr/lib64/icinga2/sbin/icinga2 012c3000-012c9000 rw-p 00000000 00:00 0 014c2000-019a8000 rw-p 00cc2000 f9:00 3039415 /usr/lib64/icinga2/sbin/icinga2 026a1000-02b0d000 rw-p 00000000 00:00 0 [heap] 02b0d000-02b31000 rw-p 00000000 00:00 0 [heap] 36aa400000-36aa420000 r-xp 00000000 f9:00 1179657 /lib64/ld-2.12.so 36aa620000-36aa621000 r--p 00020000 f9:00 1179657 /lib64/ld-2.12.so 36aa621000-36aa622000 rw-p 00021000 f9:00 1179657 /lib64/ld-2.12.so 36aa622000-36aa623000 rw-p 00000000 00:00 0 36aa800000-36aa98a000 r-xp 00000000 f9:00 1179663 /lib64/libc-2.12.so 36aa98a000-36aab8a000 ---p 0018a000 f9:00 1179663 /lib64/libc-2.12.so 36aab8a000-36aab8e000 r--p 0018a000 f9:00 1179663 /lib64/libc-2.12.so 36aab8e000-36aab90000 rw-p 0018e000 f9:00 1179663 /lib64/libc-2.12.so 36aab90000-36aab94000 rw-p 00000000 00:00 0 36aac00000-36aac17000 r-xp 00000000 f9:00 1179664 /lib64/libpthread-2.12.so 36aac17000-36aae17000 ---p 00017000 f9:00 1179664 /lib64/libpthread-2.12.so 36aae17000-36aae18000 r--p 00017000 f9:00 1179664 /lib64/libpthread-2.12.so 36aae18000-36aae19000 rw-p 00018000 f9:00 1179664 /lib64/libpthread-2.12.so 36aae19000-36aae1d000 rw-p 00000000 00:00 0 36ab000000-36ab002000 r-xp 00000000 f9:00 1179673 /lib64/libdl-2.12.so 36ab002000-36ab202000 ---p 00002000 f9:00 1179673 /lib64/libdl-2.12.so 36ab202000-36ab203000 r--p 00002000 f9:00 1179673 /lib64/libdl-2.12.so 36ab203000-36ab204000 rw-p 00003000 f9:00 1179673 /lib64/libdl-2.12.so 36ab400000-36ab407000 r-xp 00000000 f9:00 1179667 /lib64/librt-2.12.so 36ab407000-36ab606000 ---p 00007000 f9:00 1179667 /lib64/librt-2.12.so 36ab606000-36ab607000 r--p 00006000 f9:00 1179667 /lib64/librt-2.12.so 36ab607000-36ab608000 rw-p 00007000 f9:00 1179667 /lib64/librt-2.12.so 36ab800000-36ab883000 r-xp 00000000 f9:00 1179670 /lib64/libm-2.12.so 36ab883000-36aba82000 ---p 00083000 f9:00 1179670 /lib64/libm-2.12.so 36aba82000-36aba83000 r--p 00082000 f9:00 1179670 /lib64/libm-2.12.so 36aba83000-36aba84000 rw-p 00083000 f9:00 1179670 /lib64/libm-2.12.so 36abc00000-36abc15000 r-xp 00000000 f9:00 1179672 /lib64/libz.so.1.2.3 36abc15000-36abe14000 ---p 00015000 f9:00 1179672 /lib64/libz.so.1.2.3 36abe14000-36abe15000 r--p 00014000 f9:00 1179672 /lib64/libz.so.1.2.3 36abe15000-36abe16000 rw-p 00015000 f9:00 1179672 /lib64/libz.so.1.2.3 36ac400000-36ac41d000 r-xp 00000000 f9:00 1179689 /lib64/libselinux.so.1 36ac41d000-36ac61c000 ---p 0001d000 f9:00 1179689 /lib64/libselinux.so.1 36ac61c000-36ac61d000 r--p 0001c000 f9:00 1179689 /lib64/libselinux.so.1 36ac61d000-36ac61e000 rw-p 0001d000 f9:00 1179689 /lib64/libselinux.so.1 36ac61e000-36ac61f000 rw-p 00000000 00:00 0 36ac800000-36ac816000 r-xp 00000000 f9:00 1179687 /lib64/libresolv-2.12.so 36ac816000-36aca16000 ---p 00016000 f9:00 1179687 /lib64/libresolv-2.12.so 36aca16000-36aca17000 r--p 00016000 f9:00 1179687 /lib64/libresolv-2.12.so 36aca17000-36aca18000 rw-p 00017000 f9:00 1179687 /lib64/libresolv-2.12.so 36aca18000-36aca1a000 rw-p 00000000 00:00 0 36b4400000-36b441d000 r-xp 00000000 f9:00 1179758 /lib64/libtinfo.so.5.7 36b441d000-36b461c000 ---p 0001d000 f9:00 1179758 /lib64/libtinfo.so.5.7 36b461c000-36b4620000 rw-p 0001c000 f9:00 1179758 /lib64/libtinfo.so.5.7 36b4620000-36b4621000 rw-p 00000000 00:00 0 36b6400000-36b6416000 r-xp 00000000 f9:00 1179691 /lib64/libgcc_s-4.4.7-20120601.so.1 36b6416000-36b6615000 ---p 00016000 f9:00 1179691 /lib64/libgcc_s-4.4.7-20120601.so.1 36b6615000-36b6616000 rw-p 00015000 f9:00 1179691 /lib64/libgcc_s-4.4.7-20120601.so.1 36b7c00000-36b7ce8000 r-xp 00000000 f9:00 3018593 /usr/lib64/libstdc++.so.6.0.13 36b7ce8000-36b7ee8000 ---p 000e8000 f9:00 3018593 /usr/lib64/libstdc++.so.6.0.13 36b7ee8000-36b7eef000 r--p 000e8000 f9:00 3018593 /usr/lib64/libstdc++.so.6.0.13 36b7eef000-36b7ef1000 rw-p 000ef000 f9:00 3018593 /usr/lib64/libstdc++.so.6.0.13 36b7ef1000-36b7f06000 rw-p 00000000 00:00 0 36b8c00000-36b8c03000 r-xp 00000000 f9:00 1179744 /lib64/libcom_err.so.2.1 36b8c03000-36b8e02000 ---p 00003000 f9:00 1179744 /lib64/libcom_err.so.2.1 36b8e02000-36b8e03000 r--p 00002000 f9:00 1179744 /lib64/libcom_err.so.2.1 36b8e03000-36b8e04000 rw-p 00003000 f9:00 1179744 /lib64/libcom_err.so.2.1 36b9c00000-36b9dba000 r-xp 00000000 f9:00 3021101 /usr/lib64/libcrypto.so.1.0.1e 36b9dba000-36b9fba000 ---p 001ba000 f9:00 3021101 /usr/lib64/libcrypto.so.1.0.1e 36b9fba000-36b9fd5000 r--p 001ba000 f9:00 3021101 /usr/lib64/libcrypto.so.1.0.1e 36b9fd5000-36b9fe1000 rw-p 001d5000 f9:00 3021101 /usr/lib64/libcrypto.so.1.0.1e 36b9fe1000-36b9fe5000 rw-p 00000000 00:00 0 36ba400000-36ba40a000 r-xp 00000000 f9:00 1179741 /lib64/libkrb5support.so.0.1 36ba40a000-36ba609000 ---p 0000a000 f9:00 1179741 /lib64/libkrb5support.so.0.1 36ba609000-36ba60a000 r--p 00009000 f9:00 1179741 /lib64/libkrb5support.so.0.1 36ba60a000-36ba60b000 rw-p 0000a000 f9:00 1179741 /lib64/libkrb5support.so.0.1 36ba800000-36ba802000 r-xp 00000000 f9:00 1179737 /lib64/libkeyutils.so.1.3 36ba802000-36baa01000 ---p 00002000 f9:00 1179737 /lib64/libkeyutils.so.1.3 36baa01000-36baa02000 r--p 00001000 f9:00 1179737 /lib64/libkeyutils.so.1.3 36baa02000-36baa03000 rw-p 00002000 f9:00 1179737 /lib64/libkeyutils.so.1.3 36bac00000-36bacdc000 r-xp 00000000 f9:00 1179745 /lib64/libkrb5.so.3.3 36bacdc000-36baedb000 ---p 000dc000 f9:00 1179745 /lib64/libkrb5.so.3.3 36baedb000-36baee5000 r--p 000db000 f9:00 1179745 /lib64/libkrb5.so.3.3 36baee5000-36baee7000 rw-p 000e5000 f9:00 1179745 /lib64/libkrb5.so.3.3 36bb000000-36bb029000 r-xp 00000000 f9:00 1179742 /lib64/libk5crypto.so.3.1 36bb029000-36bb229000 ---p 00029000 f9:00 1179742 /lib64/libk5crypto.so.3.1 36bb229000-36bb22a000 r--p 00029000 f9:00 1179742 /lib64/libk5crypto.so.3.1 36bb22a000-36bb22b000 rw-p 0002a000 f9:00 1179742 /lib64/libk5crypto.so.3.1 36bb22b000-36bb22c000 rw-p 00000000 00:00 0 36bb400000-36bb441000 r-xp 00000000 f9:00 1179746 /lib64/libgssapi_krb5.so.2.2 36bb441000-36bb641000 ---p 00041000 f9:00 1179746 /lib64/libgssapi_krb5.so.2.2 36bb641000-36bb642000 r--p 00041000 f9:00 1179746 /lib64/libgssapi_krb5.so.2.2 36bb642000-36bb644000 rw-p 00042000 f9:00 1179746 /lib64/libgssapi_krb5.so.2.2 36bcc00000-36bcc62000 r-xp 00000000 f9:00 3021233 /usr/lib64/libssl.so.1.0.1e 36bcc62000-36bce62000 ---p 00062000 f9:00 3021233 /usr/lib64/libssl.so.1.0.1e 36bce62000-36bce66000 r--p 00062000 f9:00 3021233 /usr/lib64/libssl.so.1.0.1e 36bce66000-36bce6c000 rw-p 00066000 f9:00 3021233 /usr/lib64/libssl.so.1.0.1e 3aa5200000-3aa5261000 r-xp 00000000 f9:00 3043159 /usr/lib64/libboost_program_options.so.1.53.0 3aa5261000-3aa5460000 ---p 00061000 f9:00 3043159 /usr/lib64/libboost_program_options.so.1.53.0 3aa5460000-3aa5465000 rw-p 00060000 f9:00 3043159 /usr/lib64/libboost_program_options.so.1.53.0 3aa5600000-3aa654b000 r-xp 00000000 f9:00 3043160 /usr/lib64/libicudata.so.42.1 3aa654b000-3aa674a000 ---p 00f4b000 f9:00 3043160 /usr/lib64/libicudata.so.42.1 3aa674a000-3aa674b000 rw-p 00f4a000 f9:00 3043160 /usr/lib64/libicudata.so.42.1 3aa7000000-3aa7016000 r-xp 00000000 f9:00 3043158 /usr/lib64/libboost_thread.so.1.53.0 3aa7016000-3aa7215000 ---p 00016000 f9:00 3043158 /usr/lib64/libboost_thread.so.1.53.0 3aa7215000-3aa7217000 rw-p 00015000 f9:00 3043158 /usr/lib64/libboost_thread.so.1.53.0 3aa7800000-3aa7803000 r-xp 00000000 f9:00 3043157 /usr/lib64/libboost_system.so.1.53.0 3aa7803000-3aa7a02000 ---p 00003000 f9:00 3043157 /usr/lib64/libboost_system.so.1.53.0 3aa7a02000-3aa7a03000 rw-p 00002000 f9:00 3043157 /usr/lib64/libboost_system.so.1.53.0 3aa8800000-3aa893f000 r-xp 00000000 f9:00 3043161 /usr/lib64/libicuuc.so.42.1 3aa893f000-3aa8b3f000 ---p 0013f000 f9:00 3043161 /usr/lib64/libicuuc.so.42.1 3aa8b3f000-3aa8b50000 rw-p 0013f000 f9:00 3043161 /usr/lib64/libicuuc.so.42.1 3aa8b50000-3aa8b52000 rw-p 00000000 00:00 0 3aa9000000-3aa9188000 r-xp 00000000 f9:00 3043162 /usr/lib64/libicui18n.so.42.1 3aa9188000-3aa9388000 ---p 00188000 f9:00 3043162 /usr/lib64/libicui18n.so.42.1 3aa9388000-3aa9395000 rw-p 00188000 f9:00 3043162 /usr/lib64/libicui18n.so.42.1 3aa9395000-3aa9396000 rw-p 00000000 00:00 0 3aa9400000-3aa950e000 r-xp 00000000 f9:00 3043163 /usr/lib64/libboost_regex.so.1.53.0 3aa950e000-3aa970d000 ---p 0010e000 f9:00 3043163 /usr/lib64/libboost_regex.so.1.53.0 3aa970d000-3aa9714000 rw-p 0010d000 f9:00 3043163 /usr/lib64/libboost_regex.so.1.53.0 3aa9800000-3aa9829000 r-xp 00000000 f9:00 3041095 /usr/lib64/libedit.so.0.0.27 3aa9829000-3aa9a29000 ---p 00029000 f9:00 3041095 /usr/lib64/libedit.so.0.0.27 3aa9a29000-3aa9a2c000 rw-p 00029000 f9:00 3041095 /usr/lib64/libedit.so.0.0.27 3aa9a2c000-3aa9a2f000 rw-p 00000000 00:00 0 3aa9c00000-3aa9c22000 r-xp 00000000 f9:00 1179655 /lib64/libncurses.so.5.7 3aa9c22000-3aa9e21000 ---p 00022000 f9:00 1179655 /lib64/libncurses.so.5.7 3aa9e21000-3aa9e22000 rw-p 00021000 f9:00 1179655 /lib64/libncurses.so.5.7 7fcde8000000-7fcde8021000 rw-p 00000000 00:00 0 7fcde8021000-7fcdec000000 ---p 00000000 00:00 0 7fcdec000000-7fcdec021000 rw-p 00000000 00:00 0 7fcdec021000-7fcdf0000000 ---p 00000000 00:00 0 7fcdf0000000-7fcdf0021000 rw-p 00000000 00:00 0 7fcdf0021000-7fcdf4000000 ---p 00000000 00:00 0 7fcdf4000000-7fcdf4023000 rw-p 00000000 00:00 0 7fcdf4023000-7fcdf8000000 ---p 00000000 00:00 0 7fcdf8000000-7fcdf8021000 rw-p 00000000 00:00 0 7fcdf8021000-7fcdfc000000 ---p 00000000 00:00 0 7fcdfc000000-7fcdfc021000 rw-p 00000000 00:00 0 7fcdfc021000-7fce00000000 ---p 00000000 00:00 0 7fce00000000-7fce000bb000 rw-p 00000000 00:00 0 7fce000bb000-7fce000c6000 rw-p 00000000 00:00 0 7fce000c6000-7fce04000000 ---p 00000000 00:00 0 7fce04000000-7fce04026000 rw-p 00000000 00:00 0 7fce04026000-7fce08000000 ---p 00000000 00:00 0 7fce08000000-7fce08021000 rw-p 00000000 00:00 0 7fce08021000-7fce0c000000 ---p 00000000 00:00 0 7fce0c000000-7fce0c0b7000 rw-p 00000000 00:00 0 7fce0c0b7000-7fce0c0da000 rw-p 00000000 00:00 0 7fce0c0da000-7fce10000000 ---p 00000000 00:00 0 7fce10000000-7fce10021000 rw-p 00000000 00:00 0 7fce10021000-7fce14000000 ---p 00000000 00:00 0 7fce14000000-7fce14021000 rw-p 00000000 00:00 0 7fce14021000-7fce14024000 rw-p 00000000 00:00 0 7fce14024000-7fce18000000 ---p 00000000 00:00 0 7fce18000000-7fce18021000 rw-p 00000000 00:00 0 7fce18021000-7fce18029000 rw-p 00000000 00:00 0 7fce18029000-7fce1c000000 ---p 00000000 00:00 0 7fce1c000000-7fce1c021000 rw-p 00000000 00:00 0 7fce1c021000-7fce1c0a9000 rw-p 00000000 00:00 0 7fce1c0a9000-7fce20000000 ---p 00000000 00:00 0 7fce20000000-7fce20021000 rw-p 00000000 00:00 0 7fce20021000-7fce20025000 rw-p 00000000 00:00 0 7fce20025000-7fce24000000 ---p 00000000 00:00 0 7fce275af000-7fce275b0000 ---p 00000000 00:00 0 7fce275b0000-7fce275f0000 rw-p 00000000 00:00 0 [stack:14773] 7fce275f0000-7fce275f1000 ---p 00000000 00:00 0 7fce275f1000-7fce27631000 rw-p 00000000 00:00 0 [stack:43747] 7fce27631000-7fce27632000 ---p 00000000 00:00 0 7fce27632000-7fce27672000 rw-p 00000000 00:00 0 [stack:43746] 7fce27672000-7fce27673000 ---p 00000000 00:00 0 7fce27673000-7fce276b3000 rw-p 00000000 00:00 0 [stack:43745] 7fce276b3000-7fce276b4000 ---p 00000000 00:00 0 7fce276b4000-7fce276f4000 rw-p 00000000 00:00 0 [stack:43744] 7fce276f4000-7fce276f5000 ---p 00000000 00:00 0 7fce276f5000-7fce27735000 rw-p 00000000 00:00 0 [stack:43743] 7fce27735000-7fce27736000 ---p 00000000 00:00 0 7fce27736000-7fce27776000 rw-p 00000000 00:00 0 [stack:43742] 7fce27776000-7fce27777000 ---p 00000000 00:00 0 7fce27777000-7fce277b7000 rw-p 00000000 00:00 0 [stack:43741] 7fce277b7000-7fce277b8000 ---p 00000000 00:00 0 7fce277b8000-7fce277f8000 rw-p 00000000 00:00 0 [stack:43740] 7fce277f8000-7fce277f9000 ---p 00000000 00:00 0 7fce277f9000-7fce27839000 rw-p 00000000 00:00 0 [stack:43739] 7fce27839000-7fce2783a000 ---p 00000000 00:00 0 7fce2783a000-7fce2787a000 rw-p 00000000 00:00 0 [stack:43738] 7fce2787a000-7fce2787b000 ---p 00000000 00:00 0 7fce2787b000-7fce278bb000 rw-p 00000000 00:00 0 [stack:43737] 7fce278bb000-7fce278bc000 ---p 00000000 00:00 0 7fce278bc000-7fce278fc000 rw-p 00000000 00:00 0 [stack:43736] 7fce278fc000-7fce278fd000 ---p 00000000 00:00 0 7fce278fd000-7fce2793d000 rw-p 00000000 00:00 0 [stack:43735] 7fce2793d000-7fce2793e000 ---p 00000000 00:00 0 7fce2793e000-7fce2797e000 rw-p 00000000 00:00 0 7fce2797e000-7fce2797f000 ---p 00000000 00:00 0 7fce2797f000-7fce279bf000 rw-p 00000000 00:00 0 [stack:43732] 7fce279bf000-7fce279c0000 ---p 00000000 00:00 0 7fce279c0000-7fce27a00000 rw-p 00000000 00:00 0 [stack:43731] 7fce27a00000-7fce27a01000 ---p 00000000 00:00 0 7fce27a01000-7fce27a41000 rw-p 00000000 00:00 0 [stack:43730] 7fce27a41000-7fce27a42000 ---p 00000000 00:00 0 7fce27a42000-7fce27a82000 rw-p 00000000 00:00 0 [stack:43728] 7fce27a82000-7fce27a83000 ---p 00000000 00:00 0 7fce27a83000-7fce27ac3000 rw-p 00000000 00:00 0 [stack:43725] 7fce27ac3000-7fce27ac4000 ---p 00000000 00:00 0 7fce27ac4000-7fce27b04000 rw-p 00000000 00:00 0 [stack:43727] 7fce27b04000-7fce27b05000 ---p 00000000 00:00 0 7fce27b05000-7fce27b45000 rw-p 00000000 00:00 0 [stack:43729] 7fce27b45000-7fce27b46000 ---p 00000000 00:00 0 7fce27b46000-7fce27b86000 rw-p 00000000 00:00 0 [stack:43722] 7fce27b86000-7fce27b87000 ---p 00000000 00:00 0 7fce27b87000-7fce27bc7000 rw-p 00000000 00:00 0 [stack:43718] 7fce27bc7000-7fce27bd3000 r-xp 00000000 f9:00 1181075 /lib64/libnss_files-2.12.so 7fce27bd3000-7fce27dd2000 ---p 0000c000 f9:00 1181075 /lib64/libnss_files-2.12.so 7fce27dd2000-7fce27dd3000 r--p 0000b000 f9:00 1181075 /lib64/libnss_files-2.12.so 7fce27dd3000-7fce27dd4000 rw-p 0000c000 f9:00 1181075 /lib64/libnss_files-2.12.so 7fce27dd4000-7fce27dd5000 ---p 00000000 00:00 0 7fce27dd5000-7fce27e15000 rw-p 00000000 00:00 0 [stack:43721] 7fce27e15000-7fce27e16000 ---p 00000000 00:00 0 7fce27e16000-7fce27e56000 rw-p 00000000 00:00 0 [stack:43720] 7fce27e56000-7fce27e57000 ---p 00000000 00:00 0 7fce27e57000-7fce27ea5000 rw-p 00000000 00:00 0 [stack:43719] 7fce27eb1000-7fce27eb5000 rw-p 00000000 00:00 0 7fce27eb5000-7fce27eb7000 rw-p 00000000 00:00 0 7ffff1d64000-7ffff1d85000 rw-p 00000000 00:00 0 [stack] 7ffff1dc9000-7ffff1dcb000 r--p 00000000 00:00 0 [vvar] 7ffff1dcb000-7ffff1dcd000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Caught SIGABRT. Current time: 2018-08-20 12:42:07 -0500
cat /var/log/icinga2/crash/report.1534786927.216873 Application information: Application version: r2.9.0-1 Installation root: /usr Sysconf directory: /etc Run directory: /var/run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /var/run/icinga2/icinga2.pid
System information: Platform: Oracle Linux Server Platform version: 6.9 Kernel: Linux Kernel version: 4.1.12-103.9.4.el6uek.x86_64 Architecture: x86_64
Build information: Compiler: GNU 4.8.2 Build host: 5009d75245b1 Stacktrace:
[New LWP 51877] [New LWP 14773] [New LWP 43747] [New LWP 43746] [New LWP 43745] [New LWP 43744] [New LWP 43743] [New LWP 43742] [New LWP 43741] [New LWP 43740] [New LWP 43739] [New LWP 43738] [New LWP 43737] [New LWP 43736] [New LWP 43735] [New LWP 43732] [New LWP 43731] [New LWP 43730] [New LWP 43729] [New LWP 43728] [New LWP 43727] [New LWP 43725] [New LWP 43722] [New LWP 43721] [New LWP 43720] [New LWP 43719] [New LWP 43718] [Thread debugging using libthread_db enabled] 0x00000036aa8acc4d in nanosleep () from /lib64/libc.so.6
Thread 28 (Thread 0x7fce27bc6700 (LWP 43718)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000c94be2 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 27 (Thread 0x7fce27e96700 (LWP 43719)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000c94be2 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 26 (Thread 0x7fce27e55700 (LWP 43720)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000c94be2 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 25 (Thread 0x7fce27e14700 (LWP 43721)):
0 0x00000036aac0f37d in waitpid () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cd7ba6 in icinga::Application::AttachDebugger(icinga::String const&, bool) ()
No symbol table info available.
2 0x0000000000c3d6fa in icinga::Application::SigAbrtHandler(int) ()
No symbol table info available.
3
No symbol table info available.
4 0x00000036aa832495 in raise () from /lib64/libc.so.6
No symbol table info available.
5 0x00000036aa833c75 in abort () from /lib64/libc.so.6
No symbol table info available.
6 0x00000036aa8703a7 in __libc_message () from /lib64/libc.so.6
No symbol table info available.
7 0x00000036aa875dee in malloc_printerr () from /lib64/libc.so.6
No symbol table info available.
8 0x00000036aa878c80 in _int_free () from /lib64/libc.so.6
No symbol table info available.
9 0x0000000000c1d7dd in boost::detail::function::functor_manager<boost::algorithm::detail::token_finderF<boost::algori thm::detail::is_any_ofF > >::manage(boost::detail::function::function_buffer const&, boost::detail::function::func tion_buffer&, boost::detail::function::functor_manager_operation_type) ()
No symbol table info available.
10 0x0000000000bfdda2 in ?? ()
No symbol table info available.
11 0x0000000000bfe558 in icinga::String::Split(char const*) const ()
No symbol table info available.
12 0x0000000000d7a28a in icinga::PluginUtility::ParseCheckOutput(icinga::String const&) ()
No symbol table info available.
13 0x0000000000ebadc8 in icinga::PluginCheckTask::ProcessFinishedHandler(boost::intrusive_ptr const& , boost::intrusive_ptr const&, icinga::Value const&, icinga::ProcessResult const&) ()
No symbol table info available.
14 0x0000000000c94d35 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
15 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
16 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
17 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 24 (Thread 0x7fce27b85700 (LWP 43722)):
0 0x00000036aac0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cca0fd in ?? ()
No symbol table info available.
2 0x0000000000bd80fb in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
3 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 23 (Thread 0x7fce27ac2700 (LWP 43725)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000cd98ee in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 22 (Thread 0x7fce27b03700 (LWP 43727)):
0 0x00000036aac0eb2d in accept () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000bdb8cf in icinga::Socket::Accept() ()
No symbol table info available.
2 0x0000000000bdc090 in icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr const&) ()
No symbol table info available.
3 0x0000000000af55c4 in ?? ()
No symbol table info available.
4 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
5 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
6 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 21 (Thread 0x7fce27a81700 (LWP 43728)):
0 0x00000036aac0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cca0fd in ?? ()
No symbol table info available.
2 0x0000000000eab5e1 in icinga::CheckerComponent::CheckThreadProc() ()
No symbol table info available.
3 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 20 (Thread 0x7fce27b44700 (LWP 43729)):
0 0x00000036aa8df383 in poll () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c6401a in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 19 (Thread 0x7fce27a40700 (LWP 43730)):
0 0x00000036aa8df383 in poll () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c6401a in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 18 (Thread 0x7fce279ff700 (LWP 43731)):
0 0x00000036aa8df383 in poll () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c6401a in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 17 (Thread 0x7fce279be700 (LWP 43732)):
0 0x00000036aa8df383 in poll () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c6401a in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 16 (Thread 0x7fce2793c700 (LWP 43735)):
0 0x00000036aa8e91c3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c9417e in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 15 (Thread 0x7fce278fb700 (LWP 43736)):
0 0x00000036aa8e91c3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c9417e in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 14 (Thread 0x7fce278ba700 (LWP 43737)):
0 0x00000036aa8e91c3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c9417e in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 13 (Thread 0x7fce27879700 (LWP 43738)):
0 0x00000036aa8e91c3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c9417e in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 12 (Thread 0x7fce27838700 (LWP 43739)):
0 0x00000036aa8e91c3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c9417e in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 11 (Thread 0x7fce277f7700 (LWP 43740)):
0 0x00000036aa8e91c3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c9417e in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 10 (Thread 0x7fce277b6700 (LWP 43741)):
0 0x00000036aa8e91c3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c9417e in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 9 (Thread 0x7fce27775700 (LWP 43742)):
0 0x00000036aa8e91c3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
1 0x0000000000c9417e in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
2 0x00000036b7cb6470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
3 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
4 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 8 (Thread 0x7fce27734700 (LWP 43743)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000cd98ee in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 7 (Thread 0x7fce276f3700 (LWP 43744)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000cd98ee in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 6 (Thread 0x7fce276b2700 (LWP 43745)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000cd98ee in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 5 (Thread 0x7fce27671700 (LWP 43746)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000cd98ee in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 4 (Thread 0x7fce27630700 (LWP 43747)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000cd98ee in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 3 (Thread 0x7fce275ef700 (LWP 14773)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000cd98ee in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 2 (Thread 0x7fce2797d700 (LWP 51877)):
0 0x00000036aac0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
1 0x0000000000cdf2cb in boost::condition_variable::wait(boost::unique_lock&) ()
No symbol table info available.
2 0x0000000000c94be2 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
3 0x0000003aa700c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0
No symbol table info available.
4 0x00000036aac07aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
5 0x00000036aa8e8bcd in clone () from /lib64/libc.so.6
No symbol table info available.
Thread 1 (Thread 0x7fce27e977e0 (LWP 43717)):
0 0x00000036aa8acc4d in nanosleep () from /lib64/libc.so.6
No symbol table info available.
1 0x00000036aa8acac0 in sleep () from /lib64/libc.so.6
No symbol table info available.
2 0x0000000000a27904 in icinga::Utility::Sleep(double) ()
No symbol table info available.
3 0x0000000000c65e9d in icinga::Application::RunEventLoop() ()
No symbol table info available.
4 0x0000000000c678fc in icinga::IcingaApplication::Main() ()
No symbol table info available.
5 0x0000000000c3b953 in icinga::Application::Run() ()
No symbol table info available.
6 0x0000000000d8fb1f in icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::basi c_string<char, std::char_traits, std::allocator >, std::allocator<std::basic_string<char, std::char_traits, std::allocator > > > const&) const ()
No symbol table info available.
7 0x0000000000d949ca in ?? ()
No symbol table info available.
8 0x0000000000a06607 in main ()
No symbol table info available.
Thanks, Glen