search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2.01k stars 577 forks source link

node setup: segfault on armhf #6966

Closed nbuchwitz closed 5 years ago

nbuchwitz commented 5 years ago

I've been testing the new arm builds on Raspbian stretch. The node setup ends with a segfault error, but everything seems to be correctly generated:

root@revpi-testing-stretch:/var/lib/icinga2# icinga2 node setup --ticket c8f48c7f4a0c569a09fbc1ebbb2863a678c648d0 --cn revpi-testing-stretch.sample-industries.example.org --endpoint mon01.example.org --zone revpi-testing-stretch.hrs-monitoring.ena.eu --parent_host mon01.example.org --trustedcert /var/lib/icinga2/certs/trusted-master.crt --accept-commands --accept-config
information/cli: Requesting certificate with ticket 'c8f48c7f4a0c569a09fbc1ebbb2863a678c648d0'.
information/cli: Verifying parent host connection information: host 'mon01.example.org', port '5665'.
information/cli: Using the following CN (defaults to FQDN): 'revpi-testing-stretch.sample-industries.example.org'.
information/base: Writing private key to '/var/lib/icinga2/certs//revpi-testing-stretch.sample-industries.example.org.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/certs//revpi-testing-stretch.sample-industries.example.org.crt'.
information/cli: Verifying trusted certificate file '/var/lib/icinga2/certs/trusted-master.crt'.
information/cli: Requesting a signed certificate from the parent Icinga node.
information/cli: Writing CA certificate to file '/var/lib/icinga2/certs//ca.crt'.
information/cli: Writing signed certificate to file '/var/lib/icinga2/certs//revpi-testing-stretch.sample-industries.example.org.crt'.
information/cli: Disabling the Notification feature.
warning/cli: Feature 'notification' already disabled.
information/cli: Updating the ApiListener feature.
warning/cli: Feature 'api' already enabled.
information/cli: Backup file '/etc/icinga2/features-available/api.conf.orig' already exists. Skipping backup.
information/cli: Generating zone and object configuration.
Segmentation fault

After that the service could be started and the daemon behaves also correctly.

Your Environment

icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.2-1)

Copyright (c) 2012-2018 Icinga Development Team (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Raspbian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.76-rt60-v7+
  Architecture: armv7l

Build information:
  Compiler: GNU 6.3.0
  Build host: runner-273c176c-project-297-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker mainlog
[2019-02-22 13:19:45 +0100] information/cli: Icinga application loader (version: r2.10.2-1)
[2019-02-22 13:19:45 +0100] information/cli: Loading configuration file(s).
[2019-02-22 13:19:45 +0100] information/ConfigItem: Committing config item(s).
[2019-02-22 13:19:45 +0100] information/ApiListener: My API identity: hrs-at-2019006.hrs-monitoring.ena.eu
[2019-02-22 13:19:45 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-02-22 13:19:45 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2019-02-22 13:19:45 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2019-02-22 13:19:45 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-02-22 13:19:45 +0100] information/ConfigItem: Instantiated 4 Zones.
[2019-02-22 13:19:45 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2019-02-22 13:19:45 +0100] information/ConfigItem: Instantiated 215 CheckCommands.
[2019-02-22 13:19:45 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2019-02-22 13:19:45 +0100] information/cli: Finished validating the configuration file(s).
lazyfrosch commented 5 years ago

Will have a look on it on Monday!

dnsmichi commented 5 years ago

Can you run this with gdb and attach the stacktrace?

gdb --args /usr/lib/icinga2/sbin/icinga2 node setup --ticket c8f48c7f4a0c569a09fbc1ebbb2863a678c648d0 --cn revpi-testing-stretch.sample-industries.example.org --endpoint mon01.example.org --zone revpi-testing-stretch.hrs-monitoring.ena.eu --parent_host mon01.example.org --trustedcert /var/lib/icinga2/certs/trusted-master.crt --accept-commands --accept-config

r

thread apply all bt full
nbuchwitz commented 5 years ago 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0x7487e370 (LWP 29615)]
[Thread 0x7487e370 (LWP 29615) exited]
[New Thread 0x7487e370 (LWP 29616)]
[New Thread 0x73eff370 (LWP 29617)]
[New Thread 0x734ff370 (LWP 29618)]
[New Thread 0x72cff370 (LWP 29619)]
[New Thread 0x722ff370 (LWP 29620)]
[Thread 0x72cff370 (LWP 29619) exited]
[Thread 0x734ff370 (LWP 29618) exited]
[Thread 0x73eff370 (LWP 29617) exited]
[Thread 0x7487e370 (LWP 29616) exited]
[Thread 0x722ff370 (LWP 29620) exited]
[New Thread 0x722ff370 (LWP 29621)]
[Thread 0x722ff370 (LWP 29621) exited]
[New Thread 0x722ff370 (LWP 29622)]
[New Thread 0x72cff370 (LWP 29624)]
[New Thread 0x734ff370 (LWP 29625)]
[New Thread 0x73eff370 (LWP 29626)]
[New Thread 0x7487e370 (LWP 29627)]
[Thread 0x7487e370 (LWP 29627) exited]
[Thread 0x73eff370 (LWP 29626) exited]
[Thread 0x734ff370 (LWP 29625) exited]
[Thread 0x72cff370 (LWP 29624) exited]
[Thread 0x722ff370 (LWP 29622) exited]
process 29614 is executing new program: /usr/lib/arm-linux-gnueabihf/icinga2/sbin/icinga2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0x7487e370 (LWP 29636)]
[Thread 0x7487e370 (LWP 29636) exited]
[New Thread 0x7487e370 (LWP 29637)]
[New Thread 0x746ff370 (LWP 29638)]
[New Thread 0x746bf370 (LWP 29639)]
[New Thread 0x7467f370 (LWP 29640)]
[New Thread 0x7463f370 (LWP 29641)]
[Thread 0x7463f370 (LWP 29641) exited]
[Thread 0x7467f370 (LWP 29640) exited]
[Thread 0x746bf370 (LWP 29639) exited]
[Thread 0x746ff370 (LWP 29638) exited]
[Thread 0x7487e370 (LWP 29637) exited]
[New Thread 0x7463f370 (LWP 29642)]
[Thread 0x7463f370 (LWP 29642) exited]
[New Thread 0x7463f370 (LWP 29643)]
[New Thread 0x7467f370 (LWP 29644)]
[New Thread 0x746bf370 (LWP 29645)]
[New Thread 0x746ff370 (LWP 29646)]
[New Thread 0x7487e370 (LWP 29647)]
[New Thread 0x74298370 (LWP 29656)]
[New Thread 0x74258370 (LWP 29657)]
[New Thread 0x73fff370 (LWP 29658)]
[New Thread 0x73fbf370 (LWP 29659)]
[New Thread 0x73f7f370 (LWP 29660)]
[New Thread 0x73f3f370 (LWP 29661)]
[New Thread 0x73dff370 (LWP 29662)]
[New Thread 0x73dbf370 (LWP 29663)]

Thread 1 "icinga2" received signal SIGSEGV, Segmentation fault.
std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_M_create_node<std::pair<icinga::String const, icinga::Value> const&>(std::pair<icinga::String const, icinga::Value> const&) () at /usr/include/c++/6/bits/stl_tree.h:560
560       _M_construct_node(__tmp, std::forward<_Args>(__args)...);

Thread 33 (Thread 0x73dbf370 (LWP 29663)):
#0  0x767d6294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x002311e0 in icinga::SocketEventEngineEpoll::ThreadProc(int) (this=0x0, tid=8575720) at ./lib/base/socketevents-epoll.cpp:88
        pevents = {{events = 1943791944, data = {ptr = 0x6b636f53, fd = 1801678675, u32 = 1801678675, u64 = 5713225581238382419}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 63 times>}
        ready = <optimized out>
        events = {<No data fields>}
#2  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#3  0x76d6afc4 in start_thread (arg=0x73dbf370) at pthread_create.c:335
        pd = 0x73dbf370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-775186547, -725166751, 1943794544, 2130679592, 0, 338, 2130679592, 4096, 1, 1943793324, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 32 (Thread 0x73dff370 (LWP 29662)):
#0  0x767d6294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x002311e0 in icinga::SocketEventEngineEpoll::ThreadProc(int) (this=0x0, tid=8575720) at ./lib/base/socketevents-epoll.cpp:88
        pevents = {{events = 1944054088, data = {ptr = 0x6b636f53, fd = 1801678675, u32 = 1801678675, u64 = 5713225581238382419}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 63 times>}
        ready = <optimized out>
        events = {<No data fields>}
#2  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#3  0x76d6afc4 in start_thread (arg=0x73dff370) at pthread_create.c:335
        pd = 0x73dff370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-774924403, -725166751, 1944056688, 2130679592, 0, 338, 2130679592, 4096, 1, 1944055468, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 31 (Thread 0x73f3f370 (LWP 29661)):
#0  0x767d6294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x002311e0 in icinga::SocketEventEngineEpoll::ThreadProc(int) (this=0x0, tid=8575720) at ./lib/base/socketevents-epoll.cpp:88
        pevents = {{events = 1945364808, data = {ptr = 0x6b636f53, fd = 1801678675, u32 = 1801678675, u64 = 5713225581238382419}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 63 times>}
        ready = <optimized out>
        events = {<No data fields>}
#2  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#3  0x76d6afc4 in start_thread (arg=0x73f3f370) at pthread_create.c:335
        pd = 0x73f3f370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-773613683, -725166751, 1945367408, 2130679592, 0, 338, 2130679592, 4096, 1, 1945366188, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 30 (Thread 0x73f7f370 (LWP 29660)):
#0  0x767d6294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x002311e0 in icinga::SocketEventEngineEpoll::ThreadProc(int) (this=0x0, tid=8575720) at ./lib/base/socketevents-epoll.cpp:88
        pevents = {{events = 1945626952, data = {ptr = 0x6b636f53, fd = 1801678675, u32 = 1801678675, u64 = 5713225581238382419}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 63 times>}
        ready = <optimized out>
        events = {<No data fields>}
#2  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#3  0x76d6afc4 in start_thread (arg=0x73f7f370) at pthread_create.c:335
        pd = 0x73f7f370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-773351539, -725166751, 1945629552, 2130679592, 0, 338, 2130679592, 4096, 1, 1945628332, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 29 (Thread 0x73fbf370 (LWP 29659)):
#0  0x767d6294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x002311e0 in icinga::SocketEventEngineEpoll::ThreadProc(int) (this=0x0, tid=8575720) at ./lib/base/socketevents-epoll.cpp:88
        pevents = {{events = 1945889096, data = {ptr = 0x6b636f53, fd = 1801678675, u32 = 1801678675, u64 = 5713225581238382419}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 63 times>}
        ready = <optimized out>
        events = {<No data fields>}
#2  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#3  0x76d6afc4 in start_thread (arg=0x73fbf370) at pthread_create.c:335
        pd = 0x73fbf370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-773089395, -725166751, 1945891696, 2130679592, 0, 338, 2130679592, 4096, 1, 1945890476, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 28 (Thread 0x73fff370 (LWP 29658)):
#0  0x767d6294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x002311e0 in icinga::SocketEventEngineEpoll::ThreadProc(int) (this=0x0, tid=8575720) at ./lib/base/socketevents-epoll.cpp:88
        pevents = {{events = 1946151240, data = {ptr = 0x6b636f53, fd = 1801678675, u32 = 1801678675, u64 = 5713225581238382419}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 63 times>}
        ready = <optimized out>
        events = {<No data fields>}
#2  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#3  0x76d6afc4 in start_thread (arg=0x73fff370) at pthread_create.c:335
        pd = 0x73fff370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-772827251, -725166751, 1946153840, 2130679592, 0, 338, 2130679592, 4096, 1, 1946152620, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 27 (Thread 0x74258370 (LWP 29657)):
#0  0x767d6294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x002311e0 in icinga::SocketEventEngineEpoll::ThreadProc(int) (this=0x0, tid=8575720) at ./lib/base/socketevents-epoll.cpp:88
        pevents = {{events = 1948612936, data = {ptr = 0x6b636f53, fd = 1801678675, u32 = 1801678675, u64 = 5713225581238382419}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 63 times>}
        ready = <optimized out>
        events = {<No data fields>}
#2  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#3  0x76d6afc4 in start_thread (arg=0x74258370) at pthread_create.c:335
        pd = 0x74258370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-701167731, -725166751, 1948615536, 2130679592, 0, 338, 2130679592, 4096, 1, 1948614316, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 26 (Thread 0x74298370 (LWP 29656)):
#0  0x767d6294 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x002311e0 in icinga::SocketEventEngineEpoll::ThreadProc(int) (this=0x0, tid=8575720) at ./lib/base/socketevents-epoll.cpp:88
        pevents = {{events = 1, data = {ptr = 0xd, fd = 13, u32 = 13, u64 = 13}}, {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}} <repeats 63 times>}
        ready = <optimized out>
        events = {<No data fields>}
#2  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#3  0x76d6afc4 in start_thread (arg=0x74298370) at pthread_create.c:335
        pd = 0x74298370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-700905587, -725166751, 1948877680, 2130679592, 0, 338, 2130679592, 4096, 1, 1948876460, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 25 (Thread 0x7487e370 (LWP 29647)):
#0  0x76d71ce8 in __pthread_cond_timedwait (cond=0x7c0e08 <icinga::Application::GetTP()::tp+328>, mutex=0x7c0df0 <icinga::Application::GetTP()::tp+304>, abstime=0x7487d9fc) at pthread_cond_timedwait.c:198
        _v2tmp = -1
        _a2tmp = 393
        _a2 = 393
        _v2 = -1
        _a3tmp = 45
        _a3 = 45
        _a1 = 8130060
        _nr = 240
        _v1tmp = 0
        _a4tmp = 1955060220
        _a1tmp = 8130060
        _a4 = 1955060220
        _v1 = 0
        __ret = <optimized out>
        futex_val = 45
        buffer = {__routine = 0x76d71624 <__condvar_cleanup>, __arg = 0x7487d938, __canceltype = 8125744, __prev = 0x0}
        cbuffer = {oldtype = 0, cond = 0x7c0e08 <icinga::Application::GetTP()::tp+328>, mutex = 0x7c0df0 <icinga::Application::GetTP()::tp+304>, bc_seq = 0}
        result = 0
        pshared = 0
        pi_flag = 0
        err = <optimized out>
        val = <optimized out>
        seq = <optimized out>
#1  0x001bc00c in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) (this=0x7c0df0 <icinga::Application::GetTP()::tp+304>, m=..., timeout=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:109
        guard = {m = 0x7487d9f4}
        check_for_interruption = {thread_info = 0x74700470, m = 0x7c0df0 <icinga::Application::GetTP()::tp+304>, set = false}
        the_mutex = 0x7c0df0 <icinga::Application::GetTP()::tp+304>
        cond_res = <optimized out>
#2  0x00208668 in timed_wait (this=<optimized out>, m=..., abs_time=...) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:128
No locals.
#3  timed_wait (wait_duration=..., m=..., this=<optimized out>) at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:152
        m = @0x7487d9ec: {m = 0x2f2a8b9, is_locked = 180}
#4  icinga::ThreadPool::ManagerThreadProc() (this=0x0) at ./lib/base/threadpool.cpp:239
        lock = {m = 0x7c0dd4 <icinga::Application::GetTP()::tp+276>, is_locked = false}
        total_utilization = 0
        total_pending = 0
        total_alive = 0
        total_avg_latency = 0
        now = <optimized out>
        idbuf = <incomplete type>
        lastStats = <optimized out>
#5  0x7698c9dc in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
#6  0x76d6afc4 in start_thread (arg=0x7487e370) at pthread_create.c:335
        pd = 0x7487e370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-694704243, -725166751, 1955062640, 2130702968, 0, 338, 2130702968, 4096, 8392624, 1955061420, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 24 (Thread 0x746ff370 (LWP 29646)):
#0  0x76d7194c in __pthread_cond_wait (cond=0x7c1728 <icinga::Application::GetTP()::tp+2664>, mutex=0x7c1710 <icinga::Application::GetTP()::tp+2640>) at pthread_cond_wait.c:186
        _a1 = 8132396
        _nr = 240
        _a3tmp = 1
        _a3 = 1
        _a4tmp = 0
        _a2tmp = 0
        _a2 = 128
        _a4 = 0
        __ret = <optimized out>
        futex_val = 1
        buffer = {__routine = 0x76d71624 <__condvar_cleanup>, __arg = 0x746fe950, __canceltype = 116206808, __prev = 0x0}
        cbuffer = {oldtype = 0, cond = 0x7c1728 <icinga::Application::GetTP()::tp+2664>, mutex = 0x7c1710 <icinga::Application::GetTP()::tp+2640>, bc_seq = 0}
        err = <optimized out>
        pshared = <optimized out>
        pi_flag = 0
        val = <optimized out>
        seq = <optimized out>
#1  0x001bc368 in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) (this=0x7c1710 <icinga::Application::GetTP()::tp+2640>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
        guard = {m = 0x0}
        check_for_interruption = {thread_info = 0x815638, m = 0x7c1710 <icinga::Application::GetTP()::tp+2640>, set = true}
        the_mutex = 0x7c1710 <icinga::Application::GetTP()::tp+2640>
        res = <optimized out>
#2  0x0020a0c4 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (this=0x7c17e0 <icinga::Application::GetTP()::tp+2848>, queue=...) at ./lib/base/threadpool.cpp:110
        lock = {m = 0x7c16f8 <icinga::Application::GetTP()::tp+2616>, is_locked = false}
        wi = {Callback = {_M_invoker = 0x20}, Timestamp = 0}
        st = <optimized out>
        et = <optimized out>
        latency = <optimized out>
        idbuf = <incomplete type>
        lock = {m = 0x7c16f8 <icinga::Application::GetTP()::tp+2616>, is_locked = false}
#3  0x76f6fdc8 in ?? () from /usr/lib/arm-linux-gnueabihf/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x76d6afc4 in start_thread (arg=0x746ff370) at pthread_create.c:335
        pd = 0x746ff370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-696281203, -725166751, 1953493872, 2130702608, 0, 338, 2130702608, 4096, 8425720, 1953492652, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 23 (Thread 0x746bf370 (LWP 29645)):
#0  0x76d7194c in __pthread_cond_wait (cond=0x7c1440 <icinga::Application::GetTP()::tp+1920>, mutex=0x7c1428 <icinga::Application::GetTP()::tp+1896>) at pthread_cond_wait.c:186
        _a1 = 8131652
        _nr = 240
        _a3tmp = 1
        _a3 = 1
        _a4tmp = 0
        _a2tmp = 0
        _a2 = 128
        _a4 = 0
        __ret = <optimized out>
        futex_val = 1
        buffer = {__routine = 0x76d71624 <__condvar_cleanup>, __arg = 0x746be950, __canceltype = 116206808, __prev = 0x0}
        cbuffer = {oldtype = 0, cond = 0x7c1440 <icinga::Application::GetTP()::tp+1920>, mutex = 0x7c1428 <icinga::Application::GetTP()::tp+1896>, bc_seq = 0}
        err = <optimized out>
        pshared = <optimized out>
        pi_flag = 0
        val = <optimized out>
        seq = <optimized out>
#1  0x001bc368 in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) (this=0x7c1428 <icinga::Application::GetTP()::tp+1896>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
        guard = {m = 0x0}
        check_for_interruption = {thread_info = 0x8154c8, m = 0x7c1428 <icinga::Application::GetTP()::tp+1896>, set = true}
        the_mutex = 0x7c1428 <icinga::Application::GetTP()::tp+1896>
        res = <optimized out>
#2  0x0020a0c4 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (this=0x7c14f8 <icinga::Application::GetTP()::tp+2104>, queue=...) at ./lib/base/threadpool.cpp:110
        lock = {m = 0x7c1410 <icinga::Application::GetTP()::tp+1872>, is_locked = false}
        wi = {Callback = {_M_invoker = 0x20}, Timestamp = 0}
        st = <optimized out>
        et = <optimized out>
        latency = <optimized out>
        idbuf = <incomplete type>
        lock = {m = 0x7c1410 <icinga::Application::GetTP()::tp+1872>, is_locked = false}
#3  0x76f6fdc8 in ?? () from /usr/lib/arm-linux-gnueabihf/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x76d6afc4 in start_thread (arg=0x746bf370) at pthread_create.c:335
        pd = 0x746bf370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-696543347, -725166751, 1953231728, 2130702608, 0, 338, 2130702608, 4096, 8426216, 1953230508, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 22 (Thread 0x7467f370 (LWP 29644)):
#0  0x76d7194c in __pthread_cond_wait (cond=0x7c1158 <icinga::Application::GetTP()::tp+1176>, mutex=0x7c1140 <icinga::Application::GetTP()::tp+1152>) at pthread_cond_wait.c:186
        _a1 = 8130908
        _nr = 240
        _a3tmp = 1
        _a3 = 1
        _a4tmp = 0
        _a2tmp = 0
        _a2 = 128
        _a4 = 0
        __ret = <optimized out>
        futex_val = 1
        buffer = {__routine = 0x76d71624 <__condvar_cleanup>, __arg = 0x7467e950, __canceltype = 116206808, __prev = 0x0}
        cbuffer = {oldtype = 0, cond = 0x7c1158 <icinga::Application::GetTP()::tp+1176>, mutex = 0x7c1140 <icinga::Application::GetTP()::tp+1152>, bc_seq = 0}
        err = <optimized out>
        pshared = <optimized out>
        pi_flag = 0
        val = <optimized out>
        seq = <optimized out>
#1  0x001bc368 in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) (this=0x7c1140 <icinga::Application::GetTP()::tp+1152>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
        guard = {m = 0x0}
        check_for_interruption = {thread_info = 0x815358, m = 0x7c1140 <icinga::Application::GetTP()::tp+1152>, set = true}
        the_mutex = 0x7c1140 <icinga::Application::GetTP()::tp+1152>
        res = <optimized out>
#2  0x0020a0c4 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (this=0x7c1210 <icinga::Application::GetTP()::tp+1360>, queue=...) at ./lib/base/threadpool.cpp:110
        lock = {m = 0x7c1128 <icinga::Application::GetTP()::tp+1128>, is_locked = false}
        wi = {Callback = {_M_invoker = 0x20}, Timestamp = 0}
        st = <optimized out>
        et = <optimized out>
        latency = <optimized out>
        idbuf = <incomplete type>
        lock = {m = 0x7c1128 <icinga::Application::GetTP()::tp+1128>, is_locked = false}
#3  0x76f6fdc8 in ?? () from /usr/lib/arm-linux-gnueabihf/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x76d6afc4 in start_thread (arg=0x7467f370) at pthread_create.c:335
        pd = 0x7467f370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-696805491, -725166751, 1952969584, 2130702608, 0, 338, 2130702608, 4096, 8470072, 1952968364, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 21 (Thread 0x7463f370 (LWP 29643)):
#0  0x76d7194c in __pthread_cond_wait (cond=0x7c0e70 <icinga::Application::GetTP()::tp+432>, mutex=0x7c0e58 <icinga::Application::GetTP()::tp+408>) at pthread_cond_wait.c:186
        _a1 = 8130164
        _nr = 240
        _a3tmp = 1
        _a3 = 1
        _a4tmp = 0
        _a2tmp = 0
        _a2 = 128
        _a4 = 0
        __ret = <optimized out>
        futex_val = 1
        buffer = {__routine = 0x76d71624 <__condvar_cleanup>, __arg = 0x7463e950, __canceltype = 116206808, __prev = 0x0}
        cbuffer = {oldtype = 0, cond = 0x7c0e70 <icinga::Application::GetTP()::tp+432>, mutex = 0x7c0e58 <icinga::Application::GetTP()::tp+408>, bc_seq = 0}
        err = <optimized out>
        pshared = <optimized out>
        pi_flag = 0
        val = <optimized out>
        seq = <optimized out>
#1  0x001bc368 in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) (this=0x7c0e58 <icinga::Application::GetTP()::tp+408>, m=...) at /usr/include/boost/thread/pthread/condition_variable.hpp:77
        guard = {m = 0x0}
        check_for_interruption = {thread_info = 0x815200, m = 0x7c0e58 <icinga::Application::GetTP()::tp+408>, set = true}
        the_mutex = 0x7c0e58 <icinga::Application::GetTP()::tp+408>
        res = <optimized out>
#2  0x0020a0c4 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (this=0x7c0f28 <icinga::Application::GetTP()::tp+616>, queue=...) at ./lib/base/threadpool.cpp:110
        lock = {m = 0x7c0e40 <icinga::Application::GetTP()::tp+384>, is_locked = false}
        wi = {Callback = {_M_invoker = 0x7463ea1c}, Timestamp = 3.8138506198609685e+180}
        st = <optimized out>
        et = <optimized out>
        latency = <optimized out>
        idbuf = <incomplete type>
        lock = {m = 0x7c0e40 <icinga::Application::GetTP()::tp+384>, is_locked = false}
#3  0x76f6fdc8 in ?? () from /usr/lib/arm-linux-gnueabihf/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x76d6afc4 in start_thread (arg=0x7463f370) at pthread_create.c:335
        pd = 0x7463f370
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {-697067635, -725166751, 1952707440, 2130702608, 0, 338, 2130702608, 4096, 8462112, 1952706220, 0 <repeats 54 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, 
              cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x767d5bc8 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:76 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 1 (Thread 0x76ff1220 (LWP 29614)):
#0  std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_M_create_node<std::pair<icinga::String const, icinga::Value> const&>(std::pair<icinga::String const, icinga::Value> const&) () at /usr/include/c++/6/bits/stl_tree.h:560
        __tmp = <optimized out>
#1  std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node::operator()<std::pair<icinga::String const, icinga::Value> const&>(std::pair<icinga::String const, icinga::Value> const&) const () at /usr/include/c++/6/bits/stl_tree.h:473
No locals.
#2  std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_M_insert_<std::pair<icinga::String const, icinga::Value> const&, std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::pair<icinga::String const, icinga::Value> const&, std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node&) [clone .isra.2561] [clone .lto_priv.5336] (this=this@entry=0x829b2c, 
    __x=__x@entry=0x5b4950 <std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_M_insert_unique_<std::pair<icinga::String const, icinga::Value> const&, std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node>(std::_Rb_tree_const_iterator<std::pair<icinga::String const, icinga::Value> >, std::pair<icinga::String const, icinga::Value> const&, std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node&) [clone .isra.2562] [clone .lto_priv.5703]+68>, __p=0x829b30, __p@entry=0x828f10, __v=...) at /usr/include/c++/6/bits/stl_tree.h:1535
        __insert_left = true
        __z = <optimized out>
#3  0x005b4950 in std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_M_insert_unique_<std::pair<icinga::String const, icinga::Value> const&, std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node>(std::_Rb_tree_const_iterator<std::pair<icinga::String const, icinga::Value> >, std::pair<icinga::String const, icinga::Value> const&, std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node&) [clone .isra.2562] [clone .lto_priv.5703] (
    this=this@entry=0x829b2c, __position=..., __position@entry=..., __v=...) at /usr/include/c++/6/bits/stl_tree.h:2004
        __res = {
          first = 0x5b4950 <std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_M_insert_unique_<std::pair<icinga::String const, icinga::Value> const&, std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node>(std::_Rb_tree_const_iterator<std::pair<icinga::String const, icinga::Value> >, std::pair<icinga::String const, icinga::Value> const&, std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_Alloc_node&) [clone .isra.2562] [clone .lto_priv.5703]+68>, second = 0x828f10}
#4  0x0019398c in std::_Rb_tree<icinga::String, std::pair<icinga::String const, icinga::Value>, std::_Select1st<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::_M_insert_unique<std::pair<icinga::String const, icinga::Value> const*>(std::pair<icinga::String const, icinga::Value> const*, std::pair<icinga::String const, icinga::Value> const*) () at /usr/include/c++/6/bits/stl_tree.h:2250
No locals.
#5  std::map<icinga::String, icinga::Value, std::less<icinga::String>, std::allocator<std::pair<icinga::String const, icinga::Value> > >::map(std::initializer_list<std::pair<icinga::String const, icinga::Value> >, std::less<icinga::String> const&, std::allocator<std::pair<icinga::String const, icinga::Value> > const&) (__a=..., __comp=..., __l=..., this=0x829b2c) at /usr/include/c++/6/bits/stl_map.h:215
No locals.
#6  icinga::Dictionary::Dictionary(std::initializer_list<std::pair<icinga::String const, icinga::Value> >) (this=this@entry=0x829b20, init=...) at ./lib/base/dictionary.cpp:46
No locals.
#7  0x0035f518 in icinga::NodeUtility::GenerateNodeIcingaConfig(icinga::String const&, icinga::String const&, icinga::String const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<icinga::String, std::allocator<icinga::String> > const&) (endpointName=..., zoneName=..., parentZoneName=..., endpoints=..., globalZones=...)
    at ./lib/cli/nodeutility.cpp:101
        config = {px = 0x82c628}
        myParentZoneMembers = {px = 0x8298a8}
#8  0x00365c20 in icinga::NodeSetupCommand::SetupNode(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (vm=..., ap=...) at ./lib/cli/nodesetupcommand.cpp:536
        ticket = {m_Data = <incomplete type>}
        connectToParent = 196
        parentHost = {m_Data = <incomplete type>}
        parentPort = {m_Data = <incomplete type>}
        trustedParentCert = {<No data fields>}
        cn = {m_Data = <incomplete type>}
        certsDir = {m_Data = <incomplete type>}
        user = {m_Data = <incomplete type>}
        group = {m_Data = <incomplete type>}
        key = {m_Data = <incomplete type>}
        cert = {m_Data = <incomplete type>}
        ca = {m_Data = <incomplete type>}
        apipath = {m_Data = <incomplete type>}
        fp = <incomplete type>
        tempApiPath = {m_Data = <incomplete type>}
        __PRETTY_FUNCTION__ = "static int icinga::NodeSetupCommand::SetupNode(const boost::program_options::variables_map&, const std::vector<std::__cxx11::basic_string<char> >&)"
        endpointName = {m_Data = <incomplete type>}
        zoneName = {m_Data = <incomplete type>}
        parentZoneName = {m_Data = <incomplete type>}
        globalZones = {<No data fields>}
        setupGlobalZones = {<No data fields>}
#9  0x0036c7b4 in icinga::NodeSetupCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (this=<optimized out>, vm=..., ap=...) at ./lib/cli/nodesetupcommand.cpp:109
No locals.
#10 0x001d61d8 in Main () at ./icinga-app/icinga.cpp:681
        args = {<No data fields>}
        argc = 2130698640
        argv = 0x40
        autocomplete = <optimized out>
        autoindex = 2130698564
        icingaUser = {m_Data = <incomplete type>}
        icingaGroup = {m_Data = <incomplete type>}
        visibleDesc = {m_caption = <incomplete type>, m_line_length = 80, m_min_description_length = 40, m_options = {<No data fields>}, belong_to_group = {<No data fields>}, groups = {<No data fields>}}
        hiddenDesc = {m_caption = <incomplete type>, m_line_length = 80, m_min_description_length = 40, m_options = {<No data fields>}, belong_to_group = {<No data fields>}, groups = {<No data fields>}}
        positionalDesc = {m_names = {<No data fields>}, m_trailing = <incomplete type>}
        cmdname = {m_Data = <incomplete type>}
        command = {px = 0x7f7ce0}
        vm = <incomplete type>
        rc = 1
#11 0x00178410 in main (argc=18, argv=0x7efff554) at ./icinga-app/icinga.cpp:975
        rc = <optimized out>
dnsmichi commented 5 years ago

If I am not mistaken, this looks familiar to the gcc regression we've analyzed before.

nbuchwitz commented 5 years ago

I'm afraid so. Besides the node setup nothing seems to be broken (all stretch PIs are running stable since friday)

dnsmichi commented 5 years ago

Do node wizard or pki request work?

lazyfrosch commented 5 years ago

node wizard fails in the same fashion:

(gdb) bt
#0  0x005b4824 in ?? ()
#1  0x005b4950 in ?? ()
#2  0x0019398c in icinga::Dictionary::Dictionary(std::initializer_list<std::pair<icinga::String const, icinga::Value> >) ()
#3  0x0035f518 in icinga::NodeUtility::GenerateNodeIcingaConfig(icinga::String const&, icinga::String const&, icinga::String const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<icinga::String, std::allocator<icinga::String> > const&)
    ()
#4  0x003622c8 in icinga::NodeWizardCommand::ClientSetup() const ()
#5  0x0036aa14 in icinga::NodeWizardCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const ()
#6  0x001d61d8 in ?? ()
#7  0x00178410 in main ()

I'm not sure what we can do to fix this in stretch on armhf...

lazyfrosch commented 5 years ago

pki request works fine.

madtempest commented 5 years ago

Hi folks, I realise there is plenty of issues that need attention and not enough devs or time, but is there any scope/eta/suggestions about this one please?

nbuchwitz commented 5 years ago

@madtempest: This issue isn't a show stopper as the functionality of the daemon is not affected. The node wizzard also generates a fully functional configuration, so the crash can be ignored (for now).

I'm pretty sure that the developers try their best to fix this issue - but keep in mind that this issue is caused by a optimization bug in the GCC version stretch is using and not by something that is broken in Icinga!

madtempest commented 5 years ago

@nbuchwitz Thanks for the info. Just an an FYI, when you use the command string icinga2 node setup --master --disable-confd it doesn't actually complete enough of the setup process to be functional as the config generated does not reach the step where it would disable config.

I am also at a loss as to how gcc interacts with this performing config - but I am not a developer!

Thank you again very much for the update!

dnsmichi commented 5 years ago

We could label this as "wontfix" since it really is a gcc compiler bug. Still I prefer to keep this open for transparency.

The issue is exactly the same as with #6761 but this time the unity build hack does not prevent crashes unfortunately. I see two options: 1) Wait for buster and gcc7+ 2) Try to build boost + icinga2 with gcc7 on armhf.

Al2Klimov commented 5 years ago

@dnsmichi I don't agree with you as we've already worked around compiler bugs, e.g. #7005, #7019.

Al2Klimov commented 5 years ago

@lazyfrosch Did we change anything in #7005 which could help here?

dnsmichi commented 5 years ago

@Al2Klimov this getting off-topic, but still - the only code wise solution would be to revert initializer lists and the ::value_type for vectors. This is something I am not going to allow for only affecting a single distribution, with a well known regression affecting any C++ application compiled with gcc5/6 on armhf.

Al2Klimov commented 5 years ago

Would it be an option to install a newer GCC on our build system like this:

deb http://deb.debian.org/debian stretch main
deb http://security.debian.org/debian-security stretch/updates main
deb http://deb.debian.org/debian stretch-updates main

deb http://deb.debian.org/debian buster main
deb http://security.debian.org/debian-security buster/updates main
deb http://deb.debian.org/debian buster-updates main
dnsmichi commented 5 years ago

I'd wait up until the asio/network thingy is ready, including boost package updates.

madtempest commented 5 years ago

Is it safe to suggest that this is unlikely to be resolved any time soon and I should abandon my arm-based icinga2 master instance?

lazyfrosch commented 5 years ago

@madtempest not an easy fix for us, as it is an compiler issue specific to ARM.

Do you have any specific problems apart from the CLI commands?

dnsmichi commented 5 years ago

I'm inclined to say, that's impossible to fix on our side (except for reverting everything from the past 2 years of development). A more modern compiler is needed, which you cannot get up until Debian Buster is available - and Raspbian then. @nbuchwitz told me lately that running the CLI commands twice works most of the time, so retrying on failure in a script is a good option.

dnsmichi commented 5 years ago

Since Raspbian packages for Debian Buster are on the rise, we won't invest any time nor resources into gcc regressions on armhf and Stretch. This is something where application developers cannot do anything about when the compiler's generated assembler code results in memory corruption. gcc5 on armhf is broken for the time being. Period.