Icinga2 crash: __cxa_throw in /usr/lib64/icinga2/sbin/icinga2

ulyszk commented 2 years ago

Describe the bug

Icinga has crashed:

[2022-02-05 19:00:37 +0100] information/FileLogger: 'main-log' started.
[2022-02-05 19:00:37 +0100] information/ApiListener: 'api' started.
[2022-02-05 19:00:37 +0100] information/ApiListener: Started new listener on '[::]:5665'
[2022-02-05 19:00:37 +0100] critical/cli: Exception in main process: Error: basic_ios::clear

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# std::__throw_ios_failure(char const*) in /lib64/libstdc++.so.6
 2# 0x00007F520F2B1F6A in /lib64/libstdc++.so.6
 3# icinga::Utility::SaveJsonFile(icinga::String const&, int, icinga::Value const&) in /usr/lib64/icinga2/sbin/icinga2
 4# icinga::ApiListener::UpdateStatusFile(boost::asio::ip::basic_endpoint<boost::asio::ip::tcp>) in /usr/lib64/icinga2/sbin/icinga2
 5# icinga::ApiListener::AddListener(icinga::String const&, icinga::String const&) in /usr/lib64/icinga2/sbin/icinga2
 6# icinga::ApiListener::Start(bool) in /usr/lib64/icinga2/sbin/icinga2
 7# icinga::ConfigObject::Activate(bool, icinga::Value const&) in /usr/lib64/icinga2/sbin/icinga2
 8# icinga::ConfigItem::ActivateItems(std::vector<boost::intrusive_ptr<icinga::ConfigItem>, std::allocator<boost::intrusive_ptr<icinga::ConfigItem> > > const&, bool, bool, bool, icinga::Value const&) in /usr/lib64/icinga2/sbin/icinga2
 9# 0x0000000000B81C00 in /usr/lib64/icinga2/sbin/icinga2
10# icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::string, std::allocator<std::string> > const&) const in /usr/lib64/icinga2/sbin/icinga2
11# 0x0000000000909920 in /usr/lib64/icinga2/sbin/icinga2
12# main in /usr/lib64/icinga2/sbin/icinga2
13# __libc_start_main in /lib64/libc.so.6
14# 0x0000000000906341 in /usr/lib64/icinga2/sbin/icinga2

Context:

        (0) Activating object 'api' of type 'ApiListener'

To Reproduce

Just happened, So far I have no idea why.

Expected behavior

Not to crash.

Your Environment

Include as many relevant details about the environment you experienced the problem in

Version used (icinga2 --version):


icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.2-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

System information: Platform: CentOS Linux Platform version: 7 (Core) Kernel: Linux Kernel version: 3.10.0-1160.42.2.el7.x86_64 Architecture: x86_64

Build information: Compiler: GNU 4.8.5 Build host: runner-hh8q3bz2-project-322-concurrent-0 OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017

Application information:

General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2

Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var

Internal paths: Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid


* Operating System and version:

cat /etc/centos-release CentOS Linux release 7.9.2009 (Core)

* Enabled features (`icinga2 feature list`):

icinga2 feature list Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus notification opentsdb perfdata statusdata syslog Enabled features: api checker mainlog

* Config validation (`icinga2 daemon -C`):

icinga2 daemon -C [2022-02-21 08:19:28 +0100] information/cli: Icinga application loader (version: 2.13.2-1) [2022-02-21 08:19:28 +0100] information/cli: Loading configuration file(s). [2022-02-21 08:19:28 +0100] information/ConfigItem: Committing config item(s). [2022-02-21 08:19:28 +0100] information/ApiListener: My API identity: enter-infi-test.alig.hu [2022-02-21 08:19:28 +0100] information/ConfigItem: Instantiated 1 CheckerComponent. [2022-02-21 08:19:28 +0100] information/ConfigItem: Instantiated 4 Zones. [2022-02-21 08:19:28 +0100] information/ConfigItem: Instantiated 1 IcingaApplication. [2022-02-21 08:19:28 +0100] information/ConfigItem: Instantiated 2 Endpoints. [2022-02-21 08:19:28 +0100] information/ConfigItem: Instantiated 1 FileLogger. [2022-02-21 08:19:28 +0100] information/ConfigItem: Instantiated 245 CheckCommands. [2022-02-21 08:19:28 +0100] information/ConfigItem: Instantiated 1 ApiListener. [2022-02-21 08:19:28 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2022-02-21 08:19:28 +0100] information/cli: Finished validating the configuration file(s).

julianbrost commented 2 years ago

Can you reproduce this behavior?

There must have been some error while writing /var/cache/icinga2/api-state.json (or a temporary file with some random characters appended). So please check if the /var/cache/icinga2/ directory is properly writable for the icinga user and the filesystem is not out of space or something like that.

If the problem persists, you can try to run icinga using strace (strace -f -yy icinga2 daemon) and look for lines related to the file mentioned above.

Al2Klimov commented 1 year ago

Hello again @ulyszk!

Any news from your side?

Best, AK

Icinga / icinga2