search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2.03k stars 578 forks source link

Icinga2 crashes after using some api-commands on Fedora 28 #6533

Closed caregg closed 6 years ago

caregg commented 6 years ago

I am experiencing crashes while using the icinga-api. At first I notices it, after trying to install the icinga-director, the director-kickstarter got an "empty result", because icinga crashed. But it's not necessary to install the director, the problem occurs much earlier...

Current Behavior

Some curl-commands let icinga crashing immediately. For example:

CRASH: curl -k  -u USER:PASSWORD 'https://icinga.fqdn:5665/v1/status'
CRASH: curl -k  -u USER:PASSWORD 'https://icinga.fqdn:5665/v1/objects/hosts'

This commands are working:

OK:  curl -k  -u root:PASSWORD 'https://icinga.fqdn:5665/v1'
OK:  curl -k  -u root:PASSWORD 'https://icinga.fqdn:5665/v1/objects/hosts/icinga.fqdn'

Steps to Reproduce (for bugs)

  1. dnf install icinga2
  2. icinga2 api setup
  3. service icinga2 restart
  4. curl -k -u USER:PASSWORD 'https://icinga.fqdn:5665/v1/status'

Your Environment

icinga2 - The Icinga 2 network monitoring daemon (version: r2.9.1-1)

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Fedora
  Platform version: 28 (Twenty Eight)
  Kernel: Linux
  Kernel version: 4.17.11-200.fc28.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.1.1
  Build host: unknown

Debug-Error-Message:

[2018-08-08 15:07:05 +0200] information/ApiListener: New client connection from [10.11.12.13]:43428 (no client certificate)
[2018-08-08 15:07:05 +0200] notice/ApiListener: New HTTP client
[2018-08-08 15:07:05 +0200] debug/HttpRequest: line: GET /v1/status HTTP/1.1, tokens: 3
[2018-08-08 15:07:05 +0200] information/HttpServerConnection: Request: GET /v1/status (from [10.11.12.13]:43428, user: root)
[2018-08-08 15:07:05 +0200] notice/WorkQueue: Spawning WorkQueue threads for 'HttpServerConnection'
[New Thread 0x7fffb4ff9700 (LWP 22238)]
/usr/include/c++/8/bits/unique_ptr.h:328: typename std::add_lvalue_reference<_Tp>::type std::unique_ptr<_Tp, _Dp>::operator*() const [with _Tp = icinga::Expression; _Dp = std::default_delete<icinga::Expression>; typename std::add_lvalue_reference<_Tp>::type = icinga::Expression&]: Assertion 'get() != pointer()' failed.
[Thread 0x7fffd9ffb700 (LWP 22229) exited]
Thread 35 "icinga2" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffb4ff9700 (LWP 22238)]
0x00007ffff4f05feb in raise () from /lib64/libc.so.6

debug.log from daemon-start till crash:

[2018-08-08 14:30:22 +0200] notice/WorkQueue: Spawning WorkQueue threads for 'ApiListener, RelayQueue'
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'friday' => 1533852000 -> 1533938400, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'monday' => 1534111200 -> 1534197600, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'saturday' => 1533938400 -> 1534024800, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'sunday' => 1534024800 -> 1534111200, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'thursday' => 1533765600 -> 1533852000, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'tuesday' => 1534197600 -> 1534284000, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'wednesday' => 1533679200 -> 1533765600, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'friday' => 1533852000 -> 1533938400, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'monday' => 1534111200 -> 1534197600, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'saturday' => 1533938400 -> 1534024800, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'sunday' => 1534024800 -> 1534111200, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'thursday' => 1533765600 -> 1533852000, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'tuesday' => 1534197600 -> 1534284000, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'wednesday' => 1534284000 -> 1534370400, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: Legacy timeperiod update returned 2 segments.
[2018-08-08 14:30:22 +0200] debug/TimePeriod: Removing segment 'Wed Aug  8 14:30:22 2018' <-> 'Thu Aug  9 14:30:22 2018' from TimePeriod '24x7'
[2018-08-08 14:30:22 +0200] debug/TimePeriod: Adding segment 'Wed Aug  8 00:00:00 2018' <-> 'Thu Aug  9 00:00:00 2018' to TimePeriod '24x7'
[2018-08-08 14:30:22 +0200] debug/TimePeriod: Adding segment 'Thu Aug  9 00:00:00 2018' <-> 'Fri Aug 10 00:00:00 2018' to TimePeriod '24x7'
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'friday' => 1533852000 -> 1533938400, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'monday' => 1534111200 -> 1534197600, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'thursday' => 1533765600 -> 1533852000, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'tuesday' => 1534197600 -> 1534284000, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'wednesday' => 1533679200 -> 1533765600, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'friday' => 1533852000 -> 1533938400, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'monday' => 1534111200 -> 1534197600, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'thursday' => 1533765600 -> 1533852000, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'tuesday' => 1534197600 -> 1534284000, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: ParseTimeRange: 'wednesday' => 1534284000 -> 1534370400, stride: 1
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: Legacy timeperiod update returned 2 segments.
[2018-08-08 14:30:22 +0200] debug/TimePeriod: Removing segment 'Wed Aug  8 14:30:22 2018' <-> 'Thu Aug  9 14:30:22 2018' from TimePeriod '9to5'
[2018-08-08 14:30:22 +0200] debug/TimePeriod: Adding segment 'Wed Aug  8 09:00:00 2018' <-> 'Wed Aug  8 17:00:00 2018' to TimePeriod '9to5'
[2018-08-08 14:30:22 +0200] debug/TimePeriod: Adding segment 'Thu Aug  9 09:00:00 2018' <-> 'Thu Aug  9 17:00:00 2018' to TimePeriod '9to5'
[2018-08-08 14:30:22 +0200] debug/LegacyTimePeriod: Legacy timeperiod update returned 0 segments.
[2018-08-08 14:30:22 +0200] debug/TimePeriod: Removing segment 'Wed Aug  8 14:30:22 2018' <-> 'Thu Aug  9 14:30:22 2018' from TimePeriod 'never'
[2018-08-08 14:30:22 +0200] notice/ApiListener: Relaying 'config::UpdateObject' message
[2018-08-08 14:30:22 +0200] information/ApiListener: 'api' started.
[2018-08-08 14:30:22 +0200] information/ApiListener: Adding new listener on port '5665'
[2018-08-08 14:30:22 +0200] information/CheckerComponent: 'checker' started.
[2018-08-08 14:30:22 +0200] information/NotificationComponent: 'notification' started.
[2018-08-08 14:30:22 +0200] debug/ApiListener: Not connecting to Endpoint 'icinga.fqdn' because that's us.
[2018-08-08 14:30:22 +0200] information/ConfigItem: Activated all objects.
[2018-08-08 14:30:22 +0200] notice/ApiListener: Current zone master: icinga.fqdn
[2018-08-08 14:30:22 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonCommand::Run'
[2018-08-08 14:30:22 +0200] notice/ApiListener: Connected endpoints: 
[2018-08-08 14:30:22 +0200] notice/ApiListener: Removing old log file: /var/lib/icinga2/api/log/1533731058
[2018-08-08 14:30:22 +0200] debug/IcingaApplication: In IcingaApplication::Main()
[2018-08-08 14:30:23 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.00304747%
[2018-08-08 14:30:27 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 12; Checks/s: 0
[2018-08-08 14:30:32 +0200] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.216667/s (13/min 13/5min 13/15min);
[2018-08-08 14:30:32 +0200] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2018-08-08 14:30:32 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 12; Checks/s: 0
[2018-08-08 14:30:34 +0200] debug/CheckerComponent: Scheduling info for checkable 'icinga.fqdn!disk' (2018-08-08 14:30:34 +0200): Object 'icinga.fqdn!disk', Next Check: 2018-08-08 14:30:34 +0200(1.53373e+09).
[2018-08-08 14:30:34 +0200] debug/CheckerComponent: Executing check for 'icinga.fqdn!disk'
[2018-08-08 14:30:34 +0200] debug/Checkable: Update checkable 'icinga.fqdn!disk' with check interval '60' from last check time at 2018-08-08 14:25:26 +0200 (1.53373e+09) to next check time at 2018-08-08 14:31:32 +0200(1.53373e+09).
[2018-08-08 14:30:34 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:34 +0200] notice/Process: Running command '/usr/lib64/nagios/plugins/check_disk' '-c' '10%' '-w' '20%' '-X' 'none' '-X' 'tmpfs' '-X' 'sysfs' '-X' 'proc' '-X' 'configfs' '-X' 'devtmpfs' '-X' 'devfs' '-X' 'mtmfs' '-X' 'tracefs' '-X' 'cgroup' '-X' 'fuse.gvfsd-fuse' '-X' 'fuse.gvfs-fuse-daemon' '-X' 'fdescfs' '-X' 'overlay' '-X' 'nsfs' '-m': PID 20624
[2018-08-08 14:30:34 +0200] debug/CheckerComponent: Check finished for object 'icinga.fqdn!disk'
[2018-08-08 14:30:34 +0200] notice/Process: PID 20624 ('/usr/lib64/nagios/plugins/check_disk' '-c' '10%' '-w' '20%' '-X' 'none' '-X' 'tmpfs' '-X' 'sysfs' '-X' 'proc' '-X' 'configfs' '-X' 'devtmpfs' '-X' 'devfs' '-X' 'mtmfs' '-X' 'tracefs' '-X' 'cgroup' '-X' 'fuse.gvfsd-fuse' '-X' 'fuse.gvfs-fuse-daemon' '-X' 'fdescfs' '-X' 'overlay' '-X' 'nsfs' '-m') terminated with exit code 0
[2018-08-08 14:30:34 +0200] debug/Checkable: Update checkable 'icinga.fqdn!disk' with check interval '60' from last check time at 2018-08-08 14:30:34 +0200 (1.53373e+09) to next check time at 2018-08-08 14:31:32 +0200(1.53373e+09).
[2018-08-08 14:30:34 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-08-08 14:30:34 +0200] debug/DbEvents: add checkable check history for 'icinga.fqdn!disk'
[2018-08-08 14:30:34 +0200] notice/ApiListener: Relaying 'event::CheckResult' message
[2018-08-08 14:30:37 +0200] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 12; Checks/s: 0.0166667
[2018-08-08 14:30:38 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.0418877%
[2018-08-08 14:30:38 +0200] information/ApiListener: New client connection from [10.11.12.13]:43358 (no client certificate)
[2018-08-08 14:30:38 +0200] notice/ApiListener: New HTTP client
[2018-08-08 14:30:38 +0200] debug/HttpRequest: line: GET /v1/status HTTP/1.1, tokens: 3
[2018-08-08 14:30:38 +0200] information/HttpServerConnection: Request: GET /v1/status (from [10.11.12.13]:43358, user: root)
[2018-08-08 14:30:38 +0200] notice/WorkQueue: Spawning WorkQueue threads for 'HttpServerConnection'
[2018-08-08 14:30:38 +0200] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in '/var/log/icinga2/crash/report.1533731438.652956'
caregg commented 6 years ago

Crash-Report:

  Application information:
  Application version: r2.9.1-1
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Fedora
  Platform version: 28 (Twenty Eight)
  Kernel: Linux
  Kernel version: 4.17.11-200.fc28.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.1.1
  Build host: unknown
Stacktrace:

    (0) libc.so.6: gsignal (+0x10b) [0x7f2a478affeb]
    (1) libc.so.6: abort (+0x12b) [0x7f2a4789a5c1]
    (2) /usr/lib64/icinga2/sbin/icinga2() [0x9db5d8]
    (3) icinga2: icinga::FilterUtility::GetFilterTargets(icinga::QueryDescription const&, boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::ApiUser> const&, icinga::String const&) (+0xd1c) [0x8c572c]
    (4) icinga2: icinga::StatusHandler::HandleRequest(boost::intrusive_ptr<icinga::ApiUser> const&, icinga::HttpRequest&, icinga::HttpResponse&, boost::intrusive_ptr<icinga::Dictionary> const&) (+0x2d6) [0x8abc86]
    (5) icinga2: icinga::HttpHandler::ProcessRequest(boost::intrusive_ptr<icinga::ApiUser> const&, icinga::HttpRequest&, icinga::HttpResponse&) (+0x4d7) [0x8c1c37]
    (6) icinga2: icinga::HttpServerConnection::ProcessMessageAsync(icinga::HttpRequest&, icinga::HttpResponse&, boost::intrusive_ptr<icinga::ApiUser> const&) (+0x3a) [0x8c1f5a]
    (7) icinga2: icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) (+0x32) [0x95a452]
    (8) icinga2: icinga::WorkQueue::WorkerThreadProc() (+0x965) [0x96a705]
    (9) libboost_thread.so.1.66.0: <unknown function> (+0x14af1) [0x7f2a4a364af1]
    (10) libpthread.so.0: <unknown function> (+0x7594) [0x7f2a47c3f594]
    (11) libc.so.6: clone (+0x3f) [0x7f2a479730df]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

[New LWP 22318]
[New LWP 22319]
[New LWP 22320]
[New LWP 22321]
[New LWP 22322]
[New LWP 22330]
[New LWP 22339]
[New LWP 22340]
[New LWP 22341]
[New LWP 22345]
[New LWP 22346]
[New LWP 22347]
[New LWP 22348]
[New LWP 22349]
[New LWP 22350]
[New LWP 22351]
[New LWP 22352]
[New LWP 22353]

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments

warning: Loadable section ".note.gnu.property" outside of ELF segments
0x00007f2a4793f9a0 in nanosleep () from /lib64/libc.so.6

Thread 19 (Thread 0x7f2a41811700 (LWP 22353)):
#0  0x00007f2a47c4992a in waitpid () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009c403e in icinga::Application::AttachDebugger(icinga::String const&, bool) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/application.cpp:670
        my_pid = {static NPos = 18446744073709551615, m_Data = "22291"}
        pid = <optimized out>
        __PRETTY_FUNCTION__ = "static void icinga::Application::AttachDebugger(const icinga::String&, bool)"
        status = 1969516402
        argv = <optimized out>
        my_pid_str = <optimized out>
        fd = <optimized out>
        uargv = <optimized out>
        uargv = <optimized out>
#2  0x00000000009f48d2 in icinga::Application::SigAbrtHandler(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/application.cpp:801
        sa = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
        fname = {static NPos = 18446744073709551615, m_Data = "/var/log/icinga2/crash/report.1533734120.309531"}
        dirName = {static NPos = 18446744073709551615, m_Data = "/var/log/icinga2/crash"}
        interactive_debugger = false
        ofs = <optimized out>
        trace = <optimized out>
#3  <signal handler called>
No symbol table info available.
#4  0x00007f2a478affeb in raise () from /lib64/libc.so.6
No symbol table info available.
#5  0x00007f2a4789a5c1 in abort () from /lib64/libc.so.6
No symbol table info available.
#6  0x00000000009db5d8 in std::__replacement_assert (__file=<optimized out>, __line=<optimized out>, __function=<optimized out>, __condition=<optimized out>) at /usr/include/c++/8/x86_64-redhat-linux/bits/c++config.h:2389
No locals.
#7  0x00000000008c572c in icinga::FilterUtility::GetFilterTargets(icinga::QueryDescription const&, boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::ApiUser> const&, icinga::String const&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/remote/filterutility.cpp:214
        result = std::vector of length 0, capacity 0
        provider = {px = 0x7f2a2808dde0}
        permissionFilter = 0x0
        permissionFrame = {Locals = {px = 0x7f2a2808fd30}, Self = {m_Value = {which_ = 4, storage_ = {<boost::detail::aligned_storage::aligned_storage_imp<32, 8>> = {<No data fields>}, static size = <optimized out>, static alignment = <optimized out>}}}, Sandboxed = false, Depth = 0, static m_ScriptFrames = {cleanup = {px = 0x10dc1f0, pn = {pi_ = 0x10dc210}}}, static m_Imports = {px = 0x10dc740}}
        __PRETTY_FUNCTION__ = "static std::vector<icinga::Value> icinga::FilterUtility::GetFilterTargets(const icinga::QueryDescription&, const Ptr&, const Ptr&, const icinga::String&)"
        type = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        attr = <optimized out>
        name = <optimized out>
        target = <optimized out>
        names = <optimized out>
        olock = <optimized out>
        name = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        target = <optimized out>
        type = <optimized out>
        frame = <optimized out>
        uvars = <optimized out>
        ufilter = <optimized out>
        filter_vars = <optimized out>
        filter = <optimized out>
        olock = <optimized out>
        kv = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
#8  0x00000000008abc86 in icinga::StatusHandler::HandleRequest(boost::intrusive_ptr<icinga::ApiUser> const&, icinga::HttpRequest&, icinga::HttpResponse&, boost::intrusive_ptr<icinga::Dictionary> const&) () at /usr/include/c++/8/bits/char_traits.h:285
        qd = {Types = std::set with 1 element = {[0] = {static NPos = 18446744073709551615, m_Data = "Status"}}, Provider = {px = 0x7f2a2808dde0}, Permission = {static NPos = 18446744073709551615, m_Data = "status/query"}}
        objs = std::vector of length 0, capacity 0
        result = {px = 0x1100820}
        ex = <optimized out>
#9  0x00000000008c1c37 in icinga::HttpHandler::ProcessRequest(boost::intrusive_ptr<icinga::ApiUser> const&, icinga::HttpRequest&, icinga::HttpResponse&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/remote/httphandler.cpp:109
        node = {px = 0x0}
        handlers = std::vector of length 2, capacity 2 = {{px = 0x10fb8f0}, {px = 0x1109b80}}
        path = <optimized out>
        params = {px = 0x7f2a280299c0}
        processed = false
        i = <optimized out>
        current_handlers = <optimized out>
        children = <optimized out>
        olock = <optimized out>
        current_handler = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
        handler = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        path = <optimized out>
#10 0x00000000008c1f5a in icinga::HttpServerConnection::ProcessMessageAsync(icinga::HttpRequest&, icinga::HttpResponse&, boost::intrusive_ptr<icinga::ApiUser> const&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/remote/httpserverconnection.cpp:328
        ex = <optimized out>
#11 0x000000000095a452 in std::function<void ()>::operator()() const (this=<optimized out>) at /usr/include/c++/8/bits/std_function.h:260
No locals.
#12 icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/workqueue.cpp:252
        eptr = <optimized out>
        mutex = <optimized out>
#13 0x000000000096a705 in icinga::WorkQueue::WorkerThreadProc() () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/workqueue.cpp:295
        idbuf = <incomplete type>
        lock = {m = 0x7f2a280a36d8, is_locked = false}
        task = <optimized out>
#14 0x00007f2a4a364af1 in thread_proxy () from /lib64/libboost_thread.so.1.66.0
No symbol table info available.
#15 0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#16 0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 18 (Thread 0x7f2a41852700 (LWP 22352)):
#0  0x00007f2a479733ff in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000990839 in icinga::SocketEventEngineEpoll::ThreadProc(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socketevents-epoll.cpp:88
        pevents = <optimized out>
        ready = <optimized out>
        events = <optimized out>
        lock = <optimized out>
        lock = <optimized out>
        i = <optimized out>
        event = <optimized out>
        buffer = <optimized out>
        event = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
#2  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 17 (Thread 0x7f2a41893700 (LWP 22351)):
#0  0x00007f2a479733ff in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000990839 in icinga::SocketEventEngineEpoll::ThreadProc(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socketevents-epoll.cpp:88
        pevents = <optimized out>
        ready = <optimized out>
        events = <optimized out>
        lock = <optimized out>
        lock = <optimized out>
        i = <optimized out>
        event = <optimized out>
        buffer = <optimized out>
        event = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
#2  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 16 (Thread 0x7f2a418d4700 (LWP 22350)):
#0  0x00007f2a479733ff in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000990839 in icinga::SocketEventEngineEpoll::ThreadProc(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socketevents-epoll.cpp:88
        pevents = <optimized out>
        ready = <optimized out>
        events = <optimized out>
        lock = <optimized out>
        lock = <optimized out>
        i = <optimized out>
        event = <optimized out>
        buffer = <optimized out>
        event = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
#2  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7f2a41915700 (LWP 22349)):
#0  0x00007f2a479733ff in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000990839 in icinga::SocketEventEngineEpoll::ThreadProc(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socketevents-epoll.cpp:88
        pevents = <optimized out>
        ready = <optimized out>
        events = <optimized out>
        lock = <optimized out>
        lock = <optimized out>
        i = <optimized out>
        event = <optimized out>
        buffer = <optimized out>
        event = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
#2  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 14 (Thread 0x7f2a41956700 (LWP 22348)):
#0  0x00007f2a479733ff in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000990839 in icinga::SocketEventEngineEpoll::ThreadProc(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socketevents-epoll.cpp:88
        pevents = <optimized out>
        ready = <optimized out>
        events = <optimized out>
        lock = <optimized out>
        lock = <optimized out>
        i = <optimized out>
        event = <optimized out>
        buffer = <optimized out>
        event = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
#2  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7f2a41997700 (LWP 22347)):
#0  0x00007f2a479733ff in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000990839 in icinga::SocketEventEngineEpoll::ThreadProc(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socketevents-epoll.cpp:88
        pevents = <optimized out>
        ready = <optimized out>
        events = <optimized out>
        lock = <optimized out>
        lock = <optimized out>
        i = <optimized out>
        event = <optimized out>
        buffer = <optimized out>
        event = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
#2  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7f2a419d8700 (LWP 22346)):
#0  0x00007f2a479733ff in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000990839 in icinga::SocketEventEngineEpoll::ThreadProc(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socketevents-epoll.cpp:88
        pevents = <optimized out>
        ready = <optimized out>
        events = <optimized out>
        lock = <optimized out>
        lock = <optimized out>
        i = <optimized out>
        event = <optimized out>
        buffer = <optimized out>
        event = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
#2  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7f2a41a19700 (LWP 22345)):
#0  0x00007f2a479733ff in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000000990839 in icinga::SocketEventEngineEpoll::ThreadProc(int) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socketevents-epoll.cpp:88
        pevents = <optimized out>
        ready = <optimized out>
        events = <optimized out>
        lock = <optimized out>
        lock = <optimized out>
        i = <optimized out>
        event = <optimized out>
        buffer = <optimized out>
        event = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        ex = <optimized out>
#2  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#3  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#4  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7f2a41a9b700 (LWP 22341)):
#0  0x00007f2a47c4587a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009e1aa4 in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) () at /usr/include/boost/thread/pthread/condition_variable.hpp:115
        cond_res = <optimized out>
        guard = <optimized out>
        check_for_interruption = <optimized out>
        the_mutex = <optimized out>
#2  0x0000000000767b46 in icinga::CheckerComponent::CheckThreadProc() () at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:140
        lock = {m = 0x7f2a1c0312e0, is_locked = false}
        idx = <optimized out>
        it = <optimized out>
        csi = <optimized out>
        wait = <optimized out>
        checkable = <optimized out>
        forced = <optimized out>
        check = <optimized out>
        host = <optimized out>
        service = <optimized out>
        tp = <optimized out>
        olock = <optimized out>
#3  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#4  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7f2a41adc700 (LWP 22340)):
#0  0x00007f2a47c48dc7 in accept () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x000000000098ac3b in icinga::Socket::Accept() () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/socket.cpp:328
        addr = {<No data fields>}
        addrlen = 128
        fd = <optimized out>
        __PRETTY_FUNCTION__ = "icinga::Socket::Ptr icinga::Socket::Accept()"
#2  0x00000000008e6f70 in icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr<icinga::Socket> const&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/remote/apilistener.cpp:350
        client = <optimized out>
        thread = <optimized out>
#3  0x000000000087650e in std::__invoke_impl<void, void (icinga::ApiListener::*&)(boost::intrusive_ptr<icinga::Socket> const&), icinga::ApiListener*&, boost::intrusive_ptr<icinga::TcpSocket>&> (__t=<optimized out>, __f=<optimized out>) at /usr/include/c++/8/bits/invoke.h:73
No locals.
#4  std::__invoke<void (icinga::ApiListener::*&)(boost::intrusive_ptr<icinga::Socket> const&), icinga::ApiListener*&, boost::intrusive_ptr<icinga::TcpSocket>&> (__fn=<optimized out>) at /usr/include/c++/8/bits/invoke.h:95
No locals.
#5  std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) (__args=..., this=<optimized out>) at /usr/include/c++/8/functional:400
No locals.
#6  std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>::operator()<, void>() (this=<optimized out>) at /usr/include/c++/8/functional:484
No locals.
#7  std::__invoke_impl<void, std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>>(std::__invoke_other, std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>&&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
No locals.
#8  std::__invoke<std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>>(std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)>&&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
No locals.
#9  std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=<optimized out>) at /usr/include/c++/8/thread:234
No locals.
#10 std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)> > >::operator()() (this=<optimized out>) at /usr/include/c++/8/thread:243
No locals.
#11 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (icinga::ApiListener::*(icinga::ApiListener*, boost::intrusive_ptr<icinga::TcpSocket>))(boost::intrusive_ptr<icinga::Socket> const&)> > > >::_M_run() () at /usr/include/c++/8/thread:186
No locals.
#12 0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#13 0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#14 0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7f2a4a7f3700 (LWP 22339)):
#0  0x00007f2a47c4552c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009e16ac in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) () at /usr/include/boost/thread/pthread/condition_variable.hpp:81
        res = 0
        guard = <optimized out>
        check_for_interruption = <optimized out>
        the_mutex = <optimized out>
#2  0x000000000096a80e in icinga::WorkQueue::WorkerThreadProc() () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/workqueue.cpp:280
        idbuf = <incomplete type>
        lock = {m = 0x7f2a1c010fe0, is_locked = false}
        task = <optimized out>
#3  0x00007f2a4a364af1 in thread_proxy () from /lib64/libboost_thread.so.1.66.0
No symbol table info available.
#4  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7f2a4a834700 (LWP 22330)):
#0  0x00007f2a47c4587a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009e1aa4 in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) () at /usr/include/boost/thread/pthread/condition_variable.hpp:115
        cond_res = <optimized out>
        guard = <optimized out>
        check_for_interruption = <optimized out>
        the_mutex = <optimized out>
#2  0x00000000009919cb in icinga::Timer::TimerThreadProc() () at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:140
        lock = <optimized out>
        idx = <optimized out>
        it = <optimized out>
        timer = <optimized out>
        wait = <optimized out>
        ptimer = <optimized out>
#3  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#4  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7f2a4a979700 (LWP 22322)):
#0  0x00007f2a47c4587a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009e1aa4 in boost::condition_variable::do_wait_until(boost::unique_lock<boost::mutex>&, timespec const&) () at /usr/include/boost/thread/pthread/condition_variable.hpp:115
        cond_res = <optimized out>
        guard = <optimized out>
        check_for_interruption = <optimized out>
        the_mutex = <optimized out>
#2  0x000000000098ef3b in icinga::ThreadPool::ManagerThreadProc() () at /usr/include/boost/thread/pthread/condition_variable_fwd.hpp:140
        idbuf = <incomplete type>
        lastStats = 1533734114.7842159
        total_pending = <optimized out>
        total_alive = <optimized out>
        total_avg_latency = <optimized out>
        total_utilization = <optimized out>
        now = <optimized out>
        lock = <optimized out>
        queue = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        pending = <optimized out>
        alive = <optimized out>
        avg_latency = <optimized out>
        utilization = <optimized out>
        lock = <optimized out>
        thread = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        thread = <optimized out>
        __for_range = <optimized out>
        __for_begin = <optimized out>
        __for_end = <optimized out>
        wthreads = <optimized out>
        tthreads = <optimized out>
        i = <optimized out>
        i = <optimized out>
#3  0x00007f2a484c2523 in execute_native_thread_routine () from /lib64/libstdc++.so.6
No symbol table info available.
#4  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7f2a4a938700 (LWP 22321)):
#0  0x00007f2a47c4552c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009e16ac in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) () at /usr/include/boost/thread/pthread/condition_variable.hpp:81
        res = 0
        guard = <optimized out>
        check_for_interruption = <optimized out>
        the_mutex = <optimized out>
#2  0x000000000098f87d in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/threadpool.cpp:110
        idbuf = <incomplete type>
        lock = {m = 0xdf5ad0 <icinga::Application::GetTP()::tp+2992>, is_locked = false}
        wi = <optimized out>
        st = <optimized out>
        et = <optimized out>
        latency = <optimized out>
        lock = <optimized out>
        ex = <optimized out>
        lock = <optimized out>
#3  0x00007f2a4a364af1 in thread_proxy () from /lib64/libboost_thread.so.1.66.0
No symbol table info available.
#4  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7f2a4a8f7700 (LWP 22320)):
#0  0x00007f2a47c4552c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009e16ac in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) () at /usr/include/boost/thread/pthread/condition_variable.hpp:81
        res = 0
        guard = <optimized out>
        check_for_interruption = <optimized out>
        the_mutex = <optimized out>
#2  0x000000000098f87d in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/threadpool.cpp:110
        idbuf = <incomplete type>
        lock = {m = 0xdf5790 <icinga::Application::GetTP()::tp+2160>, is_locked = false}
        wi = <optimized out>
        st = <optimized out>
        et = <optimized out>
        latency = <optimized out>
        lock = <optimized out>
        ex = <optimized out>
        lock = <optimized out>
#3  0x00007f2a4a364af1 in thread_proxy () from /lib64/libboost_thread.so.1.66.0
No symbol table info available.
#4  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7f2a4a8b6700 (LWP 22319)):
#0  0x00007f2a47c4552c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009e16ac in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) () at /usr/include/boost/thread/pthread/condition_variable.hpp:81
        res = 0
        guard = <optimized out>
        check_for_interruption = <optimized out>
        the_mutex = <optimized out>
#2  0x000000000098f87d in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/threadpool.cpp:110
        idbuf = <incomplete type>
        lock = {m = 0xdf5450 <icinga::Application::GetTP()::tp+1328>, is_locked = false}
        wi = <optimized out>
        st = <optimized out>
        et = <optimized out>
        latency = <optimized out>
        lock = <optimized out>
        ex = <optimized out>
        lock = <optimized out>
#3  0x00007f2a4a364af1 in thread_proxy () from /lib64/libboost_thread.so.1.66.0
No symbol table info available.
#4  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7f2a4a875700 (LWP 22318)):
#0  0x00007f2a47c4552c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000009e16ac in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) () at /usr/include/boost/thread/pthread/condition_variable.hpp:81
        res = 0
        guard = <optimized out>
        check_for_interruption = <optimized out>
        the_mutex = <optimized out>
#2  0x000000000098f87d in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/threadpool.cpp:110
        idbuf = <incomplete type>
        lock = {m = 0xdf5110 <icinga::Application::GetTP()::tp+496>, is_locked = false}
        wi = <optimized out>
        st = <optimized out>
        et = <optimized out>
        latency = <optimized out>
        lock = <optimized out>
        ex = <optimized out>
        lock = <optimized out>
#3  0x00007f2a4a364af1 in thread_proxy () from /lib64/libboost_thread.so.1.66.0
No symbol table info available.
#4  0x00007f2a47c3f594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#5  0x00007f2a479730df in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7f2a4a982900 (LWP 22291)):
#0  0x00007f2a4793f9a0 in nanosleep () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f2a4793f87a in sleep () from /lib64/libc.so.6
No symbol table info available.
#2  0x00000000009f54aa in icinga::Application::RunEventLoop() () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/utility.cpp:428
        lastLoop = 1533734119.5272
        now = <optimized out>
        timeDiff = <optimized out>
#3  0x00000000007f77fb in icinga::IcingaApplication::Main() () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/icinga/icingaapplication.cpp:107
No locals.
#4  0x00000000009f2e1e in icinga::Application::Run() () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/base/application.cpp:1016
        sa = {__sigaction_handler = {sa_handler = 0x9f2e60 <icinga::Application::SigUsr2Handler(int)>, sa_sigaction = 0x9f2e60 <icinga::Application::SigUsr2Handler(int)>}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
#5  0x000000000088acc5 in icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/lib/cli/daemoncommand.cpp:295
        configs = std::vector of length 1, capacity 1 = {"/etc/icinga2/icinga2.conf"}
        newItems = std::vector of length 262, capacity 512 = {{px = 0x14329f0}, {px = 0x14324b0}, {px = 0x1431f10}, {px = 0x1431570}, {px = 0x1431a70}, {px = 0x14120b0}, {px = 0x14125c0}, {px = 0x140f0d0}, {px = 0x14113a0}, {px = 0x1430f60}, {px = 0x140f580}, {px = 0x1446fa0}, {px = 0x140e6e0}, {px = 0x13dd610}, {px = 0x1116890}, {px = 0x11164c0}, {px = 0x11188a0}, {px = 0x1118cd0}, {px = 0x1411b60}, {px = 0x147afb0}, {px = 0x13b5fc0}, {px = 0x13d95b0}, {px = 0x111d970}, {px = 0x1123c70}, {px = 0x111f4f0}, {px = 0x11215f0}, {px = 0x10ff790}, {px = 0x11b68c0}, {px = 0x11a1e60}, {px = 0x111def0}, {px = 0x111e9f0}, {px = 0x111a7f0}, {px = 0x1354360}, {px = 0x111ad70}, {px = 0x111e470}, {px = 0x11b6d70}, {px = 0x1126d80}, {px = 0x13b7720}, {px = 0x11b7bd0}, {px = 0x1123170}, {px = 0x11241f0}, {px = 0x11bbfe0}, {px = 0x11bf2e0}, {px = 0x11c0e10}, {px = 0x1124770}, {px = 0x13b7160}, {px = 0x130b730}, {px = 0x11baa50}, {px = 0x11bafa0}, {px = 0x11bb520}, {px = 0x1127900}, {px = 0x1120570}, {px = 0x11c0890}, {px = 0x11b8b50}, {px = 0x1309550}, {px = 0x1309b70}, {px = 0x1120af0}, {px = 0x130c2b0}, {px = 0x130c870}, {px = 0x11c1e90}, {px = 0x13b2c00}, {px = 0x130bcf0}, {px = 0x13b5440}, {px = 0x13b5a00}, {px = 0x130aba0}, {px = 0x130a0b0}, {px = 0x13d8560}, {px = 0x13b3780}, {px = 0x1121070}, {px = 0x111c370}, {px = 0x1351340}, {px = 0x130b170}, {px = 0x13b6580}, {px = 0x1121b70}, {px = 0x13b4300}, {px = 0x13db210}, {px = 0x11267c0}, {px = 0x135ecf0}, {px = 0x1122670}, {px = 0x135edf0}, {px = 0x1124cf0}, {px = 0x1122bf0}, {px = 0x13db6a0}, {px = 0x13d9030}, {px = 0x111fff0}, {px = 0x111ef70}, {px = 0x140c6b0}, {px = 0x140e810}, {px = 0x1408da0}, {px = 0x13f5710}, {px = 0x140dd50}, {px = 0x1412990}, {px = 0x13f3bc0}, {px = 0x13f40d0}, {px = 0x13f4630}, {px = 0x140edb0}, {px = 0x13f4be0}, {px = 0x13f51c0}, {px = 0x11257f0}, {px = 0x1119cf0}, {px = 0x1119770}, {px = 0x13b31c0}, {px = 0x1127ec0}, {px = 0x11bc390}, {px = 0x11b7220}, {px = 0x13dbb50}, {px = 0x11220f0}, {px = 0x11b91b0}, {px = 0x13dc4d0}, {px = 0x12657a0}, {px = 0x13dca50}, {px = 0x1265c50}, {px = 0x13dcfd0}, {px = 0x11c2990}, {px = 0x1126200}, {px = 0x111b870}, {px = 0x13f1f00}, {px = 0x1125270}, {px = 0x11b76d0}, {px = 0x13b3d40}, {px = 0x1127340}, {px = 0x1125d70}, {px = 0x13b48c0}, {px = 0x13f2470}, {px = 0x11c2410}, {px = 0x11236f0}, {px = 0x13b6bd0}, {px = 0x11c1390}, {px = 0x111c8f0}, {px = 0x1366b90}, {px = 0x13da4c0}, {px = 0x1360340}, {px = 0x1369720}, {px = 0x13daa80}, {px = 0x13db040}, {px = 0x13d9f30}, {px = 0x111d3f0}, {px = 0x111ce70}, {px = 0x11c2f10}, {px = 0x13d8ab0}, {px = 0x111a270}, {px = 0x11bfd90}, {px = 0x13d9b30}, {px = 0x11c1910}, {px = 0x111bdf0}, {px = 0x13f29f0}, {px = 0x11bf810}, {px = 0x11c0310}, {px = 0x13f2f70}, {px = 0x111fa70}, {px = 0x13f34f0}, {px = 0x111b2f0}, {px = 0x13f3a70}, {px = 0x13b8f50}, {px = 0x13ba890}, {px = 0x13ba240}, {px = 0x13b9bf0}, {px = 0x13b95a0}, {px = 0x13baee0}, {px = 0x13bb530}, {px = 0x13b88d0}, {px = 0x13bc160}, {px = 0x13bce00}, {px = 0x13bd450}, {px = 0x13be0f0}, {px = 0x13bdaa0}, {px = 0x13c32d0}, {px = 0x13c3920}, {px = 0x13c6ba0}, {px = 0x13c6550}, {px = 0x13c71f0}, {px = 0x13c45c0}, {px = 0x13c4c10}, {px = 0x13c3f70}, {px = 0x13c7840}, {px = 0x13c5f00}, {px = 0x13c5260}, {px = 0x13c58b0}, {px = 0x13c7e90}, {px = 0x13be740}, {px = 0x13bf3e0}, {px = 0x13c06d0}, {px = 0x13c0080}, {px = 0x13bfa30}, {px = 0x13bed90}, {px = 0x13c0d20}, {px = 0x13c2660}, {px = 0x13c19c0}, {px = 0x13c2010}, {px = 0x13c1370}, {px = 0x13c8b30}, {px = 0x13c9180}, {px = 0x13ca470}, {px = 0x13cab70}, {px = 0x13cb1c0}, {px = 0x13cc490}, {px = 0x13c97d0}, {px = 0x13c9e20}, {px = 0x13cb810}, {px = 0x13cbe40}...}
        sa = {__sigaction_handler = {sa_handler = 0x871860 <SigHupHandler(int)>, sa_sigaction = 0x871860 <SigHupHandler(int)>}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0}
        runningpid = <optimized out>
        rc = <optimized out>
        start = <optimized out>
        ex = <optimized out>
        upq = <optimized out>
        errorLog = <optimized out>
#6  0x00000000009f782d in Main() () at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/icinga-app/icinga.cpp:573
        args = std::vector of length 0, capacity 0
        argc = 5
        argv = 0x7ffc357fa608
        autocomplete = <optimized out>
        autoindex = 0
        icingaUser = {static NPos = 18446744073709551615, m_Data = "icinga"}
        icingaGroup = {static NPos = 18446744073709551615, m_Data = "icinga"}
        logLevel = icinga::LogInformation
        visibleDesc = {static m_default_line_length = 80, m_caption = "Global options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 12, capacity 16 = {{px = 0x10f0b90, pn = {pi_ = 0x110fa00}}, {px = 0x110fa80, pn = {pi_ = 0x110fb20}}, {px = 0x110fb40, pn = {pi_ = 0x110fbe0}}, {px = 0x110fc50, pn = {pi_ = 0x110fd10}}, {px = 0x110fd30, pn = {pi_ = 0x110fdd0}}, {px = 0x110feb0, pn = {pi_ = 0x110ff50}}, {px = 0x110ff90, pn = {pi_ = 0x1110030}}, {px = 0x11103d0, pn = {pi_ = 0x1110450}}, {px = 0x11104d0, pn = {pi_ = 0x11105a0}}, {px = 0x11105f0, pn = {pi_ = 0x11106c0}}, {px = 0x11107a0, pn = {pi_ = 0x1110840}}, {px = 0x1110880, pn = {pi_ = 0x1110920}}}, belong_to_group = std::vector<bool> of length 12, capacity 64 = {0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1}, groups = std::vector of length 1, capacity 1 = {{px = 0x10e1ef0, pn = {pi_ = 0x1110ba0}}}}
        hiddenDesc = {static m_default_line_length = 80, m_caption = "Hidden options", m_line_length = 80, m_min_description_length = 40, m_options = std::vector of length 3, capacity 4 = {{px = 0x1110070, pn = {pi_ = 0x1110110}}, {px = 0x1110230, pn = {pi_ = 0x11102f0}}, {px = 0x1110a00, pn = {pi_ = 0x1110b00}}}, belong_to_group = std::vector<bool> of length 3, capacity 64 = {0, 0, 0}, groups = std::vector of length 0, capacity 0}
        positionalDesc = {m_names = std::vector of length 0, capacity 0, m_trailing = "arg"}
        cmdname = {static NPos = 18446744073709551615, m_Data = "daemon"}
        command = {px = 0x110ca90}
        vm = <incomplete type>
        rc = 1
#7  0x0000000000635ebd in main (argc=5, argv=0x7ffc357fa608) at /usr/src/debug/icinga2-2.9.1-1.fc28.icinga.x86_64/icinga-app/icinga.cpp:867
        rc = <optimized out>
dnsmichi commented 6 years ago

Hmmm unfortunately I'm not able to reproduce this on F28 here. The query you're firing against the API, how does it look exactly - only the URL or are there more parameters involved?

I can see that the crash sources from std::__replacement_assert which would generally mean that std::vector and likely the emplace_back() call are involved here.

Blind guess:

                        Object::Ptr target = provider->GetTargetByName(type, name);

                        if (!FilterUtility::EvaluateFilter(permissionFrame, permissionFilter, target, variableName))
                                BOOST_THROW_EXCEPTION(ScriptError("Access denied to object '" + name + "' of type '" + type + "'"));

                        result.emplace_back(std::move(target));

target may be null, but that should trigger EvaluateFilter() before. @Al2Klimov do you have experience with that, or an idea?

caregg commented 6 years ago

The crashing queries are exactly like under "Current Behavior", fired as bash-command:

curl -k -u USER:PASSWORD 'https://icinga.fqdn:5665/v1/status'
curl -k -u USER:PASSWORD 'https://icinga.fqdn:5665/v1/objects/hosts'

The second command with a hostname at the end works fine:

curl -k  -u USER:PASSWORD 'https://icinga.fqdn:5665/v1/objects/hosts/icinga.fqdn'
caregg commented 6 years ago

In the meanwhile I tried to reproduce the problem on other machines. On a Debian-server there is no problem, but on a virtual machine in VMware Player with Fedora 28 and the newest sofware-packages I've got the same error like on my real Fedora-machine:

# curl -k -u USER:PASSWORD 'https://icinga.fqdn:5665/v1/status'
curl: (52) Empty reply from server

Maybe there is a general problem with fedora at all?!

Al2Klimov commented 6 years ago

@caregg Just to be sure: You create a fresh virtual machine in VMware Player with Fedora 28 and perform your four "Steps to Reproduce"... and it reliably doesn't work?

caregg commented 6 years ago

Yes, exactly. Here is what I did:

The result is:

# curl -k -u USER:PASSWORD 'https://icinga.fqdn:5665/v1/status'
curl: (52) Empty reply from server
N-o-X commented 6 years ago

Did you use the Fedora 27 repository? Because the Fedora 28 repository on packages.icinga.com is currently not available.

I've tested this both with a manually build 2.9.1 and the Fedora 27 repository. The crash did happen on that Fedora 27 repository but not when build manually.

I'll revisit this issue when our Fedora 28 repository is up and running.

caregg commented 6 years ago

With the repository of Fedora 28.

Al2Klimov commented 6 years ago

@caregg The F28 repo seems to ship packages again. Does the error still occur with fresh F28 packages?

N-o-X commented 6 years ago

@Al2Klimov it does. Might be something packaging related.

lazyfrosch commented 6 years ago

Confirmed on a fresh Fedora 28 system with 2.9.1-1

@dnsmichi @Crunsher send me your SSH key if you want access

dnsmichi commented 6 years ago

This happens on F28 only with gcc 8.1.1 and enabled -Wp,-D_GLIBCXX_ASSERTIONS compiler flags by default during rpm builds.

https://bugzilla.redhat.com/show_bug.cgi?id=1515858 https://fedoraproject.org/wiki/Changes/HardeningFlags28

More references: https://github.com/StanfordLegion/legion/issues/350 http://lilypond.1069038.n5.nabble.com/2-19-81-aborts-on-many-ly-files-when-compiled-with-gcc8-Wp-D-GLIBCXX-ASSERTIONS-td212802.html https://sourceforge.net/p/astyle/bugs/503/ https://stackoverflow.com/questions/48381058/visual-c-why-does-stdmove-crash

I believe that std::emplace_back() with std::move() on an std::unique_ptr is a problem here. 

[2018-09-05 19:35:58 +0200] information/HttpServerConnection: Request: GET /v1/status (from [127.0.0.1]:32816, user: root)
[New Thread 0x7fffabeba700 (LWP 24247)]
/usr/include/c++/8/bits/unique_ptr.h:328: typename std::add_lvalue_reference<_Tp>::type std::unique_ptr<_Tp, _Dp>::operator*() const [with _Tp = icinga::Expression; _Dp = std::default_delete<icinga::Expression>; typename std::add_lvalue_reference<_Tp>::type = icinga::Expression&]: Assertion 'get() != pointer()' failed.
[Thread 0x7ffff0118700 (LWP 24233) exited]

Thread 68 "icinga2" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffabeba700 (LWP 24247)]
0x00007ffff4f05feb in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install icinga2-bin-2.9.1-1.fc28.icinga.x86_64
(gdb) bt full
#0  0x00007ffff4f05feb in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ffff4ef05c1 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00000000009db5d8 in std::__replacement_assert(char const*, int, char const*, char const*) ()
No symbol table info available.
#3  0x00000000008c572c in icinga::FilterUtility::GetFilterTargets(icinga::QueryDescription const&, boost::intrusive_ptr<icinga::Dictionary> const&, boost::intrusive_ptr<icinga::ApiUser> const&, icinga::String const&) ()
No symbol table info available.
#4  0x00000000008abc86 in icinga::StatusHandler::HandleRequest(boost::intrusive_ptr<icinga::ApiUser> const&, icinga::HttpRequest&, icinga::HttpResponse&, boost::intrusive_ptr<icinga::Dictionary> const&) ()
No symbol table info available.
#5  0x00000000008c1c37 in icinga::HttpHandler::ProcessRequest(boost::intrusive_ptr<icinga::ApiUser> const&, icinga::HttpRequest&, icinga::HttpResponse&) ()
No symbol table info available.
#6  0x00000000008c1f5a in icinga::HttpServerConnection::ProcessMessageAsync(icinga::HttpRequest&, icinga::HttpResponse&, boost::intrusive_ptr<icinga::ApiUser> const&) ()
No symbol table info available.
#7  0x000000000095a452 in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#8  0x000000000096a705 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#9  0x00007ffff79baaf1 in thread_proxy () from /lib64/libboost_thread.so.1.66.0
No symbol table info available.
#10 0x00007ffff5295594 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00007ffff4fc90df in clone () from /lib64/libc.so.6
No symbol table info available.
dnsmichi commented 6 years ago

For some reason, the debuginfo provided with the packages is scrambled and doesn't provide a full file table for the symbols. I've recompiled by source setup with the hardening flags and can now debug it way better.

vim debug/CMakeCache.txt

//Flags used by the CXX compiler during DEBUG builds.
CMAKE_CXX_FLAGS_DEBUG:STRING=-g -Wp,-D_GLIBCXX_ASSERTIONS

std::move is not correct, it is about the Expression being a unique_ptr.

/usr/include/c++/8/bits/unique_ptr.h:328: typename std::add_lvalue_reference<_Tp>::type std::unique_ptr<_Tp, _Dp>::operator*() const [with _Tp = icinga::Expression; _Dp = std::default_delete<icinga::Expression>; typename std::add_lvalue_reference<_Tp>::type = icinga::Expression&]: Assertion 'get() != pointer()' failed.

Thread 68 "icinga2" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff1877700 (LWP 13694)]
0x00007ffff53d9feb in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.27-30.fc28.x86_64 sssd-client-1.16.2-4.fc28.x86_64
(gdb) up
#1  0x00007ffff53c45c1 in abort () from /lib64/libc.so.6
(gdb) 
#2  0x0000000000556bf6 in std::__replacement_assert (__file=<optimized out>, __line=<optimized out>, __function=<optimized out>, __condition=<optimized out>)
    at /usr/include/c++/8/x86_64-redhat-linux/bits/c++config.h:2389
2389        __builtin_abort();
(gdb) 
#3  0x00000000007fa875 in std::unique_ptr<icinga::Expression, std::default_delete<icinga::Expression> >::operator* (this=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:328
328     __glibcxx_assert(get() != pointer());
(gdb) 
#4  0x00000000007f9ee2 in icinga::FilterUtility::GetFilterTargets (qd=..., query=..., user=..., variableName=...) at /home/michi/coding/icinga/icinga2/lib/remote/filterutility.cpp:273
273             std::ref(frame), &*ufilter, std::ref(result), variableName, _1));
(gdb)

Apparently ufilter is a nullptr which cannot be passed via reference here.

Proof: If you pass filter as parameter, everything works fine.

michi@imagine ~ $ curl -k -u root:icinga 'https://localhost:5665/v1/status?filter=abc'
{"error":404.0,"status":"No objects found."}

The fix is not the best code, but actually we need to either check for a) filter is set (and then use filter_targets) and pass the compiled filter expression, or otherwise just pass a nullptr. PR is coming for further tests.

dnsmichi commented 6 years ago

@caregg can you test the snapshot packages?

caregg commented 6 years ago

The problem disappeared in version 2.9.1-180 !!!

dnsmichi commented 6 years ago

Thanks, backported for 2.9.2 :)