Icinga 2 crashes because of segfault in libconfig.so.2.6.3

[Foxeronie]

Hi everyone,

since a few days the icinga2 daemon on one of my master hosts keeps crashing after some minutes/hours, starting Monday, 2017-08-01 04:59:10 AM Only this master (icinga-master01) is affected at this time. He stores the config files in /etc/icinga2/zones.d/ The other master is the active endpoint Both master + one other host form a galera cluster for IDO DB.

2 Masters with 2.6.3-1 on Ubuntu 14.04.5 2 Worker with 2.6.3-1 on Ubuntu 16.04.3

Backtraces: bt.txt bt_full.txt

syslog

Aug  6 07:49:50 icinga-master01 kernel: [44633.118674] icinga2[5293]: segfault at 20 ip 00007f0b0cdc3f00 sp 00007f0aeda26a00 error 4 in libconfig.so.2.6.3[7f0b0cd7f000+94000]
Aug  6 11:07:18 icinga-master01 kernel: [56492.624959] icinga2[16992]: segfault at 20 ip 00007f0ee3df6f00 sp 00007f0edcadea00 error 4 in libconfig.so.2.6.3[7f0ee3db2000+94000]
Aug  7 12:38:35 icinga-master01 kernel: [148457.928071] icinga2[23191]: segfault at 20 ip 00007f3a1f6c2f00 sp 00007f3a18c7ea00 error 4 in libconfig.so.2.6.3[7f3a1f67e000+94000]

icinga2.err has some entries:

pure virtual method called
pure virtual method called
pure virtual method called

Latest package updates:

Start-Date: 2017-07-28  06:29:56
Upgrade: apache2-data:amd64 (2.4.7-1ubuntu4.16, 2.4.7-1ubuntu4.17), apache2:amd64 (2.4.7-1ubuntu4.16, 2.4.7-1ubuntu4.17), apache2-bin:amd64 (2.4.7-1ubuntu4.16, 2.4.7-1ubuntu4.17), rsyslog:amd64 (7.4.4-1ubuntu2.6, 7.4.4-1ubuntu2.7)
End-Date: 2017-07-28  06:30:07

Start-Date: 2017-07-29  06:39:47
Upgrade: ntp:amd64 (4.2.6.p5+dfsg-3ubuntu2.14.04.11, 4.2.6.p5+dfsg-3ubuntu2.14.04.12), ntpdate:amd64 (4.2.6.p5+dfsg-3ubuntu2.14.04.11, 4.2.6.p5+dfsg-3ubuntu2.14.04.12)
End-Date: 2017-07-29  06:39:50

Start-Date: 2017-08-01  06:36:06
Upgrade: libmagickwand5:amd64 (6.7.7.10-6ubuntu3.8, 6.7.7.10-6ubuntu3.9), libmagickcore5:amd64 (6.7.7.10-6ubuntu3.8, 6.7.7.10-6ubuntu3.9), imagemagick-common:amd64 (6.7.7.10-6ubuntu3.8, 6.7.7.10-6ubuntu3.9)
End-Date: 2017-08-01  06:36:08

icinga2 --version

icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.3-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 14.04.5 LTS, Trusty Tahr
  Kernel: Linux
  Kernel version: 3.13.0-125-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.4
  Build host: lgw01-43

icinga2 feature list

Disabled features: compatlog gelf icingastatus influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command graphite ido-mysql mainlog notification

Icinga Web 2 version and modules

Version
2.4.1
Git commit
bc93f0cad45e52f57dac036960975c2c8d8fdaf0

icinga2 daemon -C

information/cli: Icinga application loader (version: r2.6.3-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: icinga-master01.domain.de
information/ConfigItem: Instantiated 6 ApiUsers.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 3 Zones.
information/ConfigItem: Instantiated 2 FileLoggers.
information/ConfigItem: Instantiated 4 Endpoints.
information/ConfigItem: Instantiated 9 NotificationCommands.
information/ConfigItem: Instantiated 8743 Notifications.
information/ConfigItem: Instantiated 119 CheckCommands.
information/ConfigItem: Instantiated 4169 Downtimes.
information/ConfigItem: Instantiated 4183 Hosts.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 28 HostGroups.
information/ConfigItem: Instantiated 445 Comments.
information/ConfigItem: Instantiated 66480 Dependencies.
information/ConfigItem: Instantiated 66 Users.
information/ConfigItem: Instantiated 66480 Services.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 7 ServiceGroups.
information/ConfigItem: Instantiated 32 ScheduledDowntimes.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 GraphiteWriter.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

zones.conf

object Endpoint "icinga-master01.domain.de" {
  host = "icinga-master01.domain.de"
  log_duration = 0
}

object Endpoint "icinga-master03.domain.de" {
  host = "icinga-master03.domain.de"
  log_duration = 0
}

object Endpoint "icinga-wn04.domain.de" {
  host = "icinga-wn04.domain.de"
  log_duration = 0
}

object Endpoint "icinga-wn05.domain.de" {
  host = "icinga-wn05.domain.de"
  log_duration = 0
}

object Zone "master" {
  endpoints = [ "icinga-master01.domain.de", "icinga-master03.domain.de" ]
}

object Zone "worker" {
  parent = "master"
  endpoints = [ "icinga-wn04.domain.de", "icinga-wn05.domain.de" ]
}

object Zone "director-global" {
  global = true
}

Best regards, Patrick

[Foxeronie]

I stopped Icinga 2 on my master01 and moved the content from /var/lib/icinga2/api/zones/ to /etc/icinga2/zones.d/ on my master03. It worked and everything was fine for the moment, but now he gets the segfaults.

[dnsmichi]

Please update to v2.7.0 and try to reproduce the issue. 2.6.x won't receive any patches anymore.

[Foxeronie]

Hi,

since updating to 2.7.0 the problem seems to be gone.