Upgrade to 2.9.0 on trusty broke mysql-ido

TM08353 commented 6 years ago

The upgrade from 2.8.4-1.trusty to 2.9.0-1.trusty seems to have broken the connection to the mysql-ido backend. Icinga2 is still running, checks are performed, notification emails sent, but no updates are written to the database and icingaweb2 is frozen at the old state.

Expected Behavior

Updates should be written to the database

Current Behavior

No updates are written

Steps to Reproduce (for bugs)

unsure - the automatic upgrade seems to have triggered this.

Context

Well, having access to the web frontend is kind of basic... ;)

Your Environment

Version used (icinga2 --version): icinga2 - The Icinga 2 network monitoring daemon (version: r2.9.0-1)

Copyright (c) 2012-2018 Icinga Development Team (https://www.icinga.com/) License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Application information: Installation root: /usr Sysconf directory: /etc Run directory: /run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid

System information: Platform: Ubuntu Platform version: 14.04.5 LTS, Trusty Tahr Kernel: Linux Kernel version: 3.13.0-153-generic Architecture: x86_64

Build information: Compiler: GNU 4.8.2 Build host: 057266a66422

Operating System and version: cat /etc/os-release NAME="Ubuntu" VERSION="14.04.5 LTS, Trusty Tahr" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 14.04.5 LTS" VERSION_ID="14.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
Enabled features (icinga2 feature list): Disabled features: api compatlog elasticsearch gelf graphite icingastatus influxdb livestatus opentsdb statusdata syslog Enabled features: checker command debuglog ido-mysql mainlog notification perfdata
Icinga Web 2 version and modules (System - About): 2.4.2 doc | 2.4.2 monitoring | 2.4.2 pnp | 1.0.1

Config validation (icinga2 daemon -C):

[2018-07-23 11:21:21 +0200] information/cli: Icinga application loader (version: r2.9.0-1)
[2018-07-23 11:21:21 +0200] information/cli: Loading configuration file(s).
[2018-07-23 11:21:22 +0200] information/ConfigItem: Committing config item(s).
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 458 Services.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 6 HostGroups.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 EventCommand.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 5 NotificationCommands.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 513 Notifications.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 54 Hosts.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 2 Downtimes.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 10 Dependencies.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 PerfdataWriter.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 3 Comments.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 3 Zones.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 259 CheckCommands.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 7 TimePeriods.
[2018-07-23 11:21:23 +0200] information/ConfigItem: Instantiated 2 Users.
[2018-07-23 11:21:23 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2018-07-23 11:21:23 +0200] information/cli: Finished validating the configuration file(s).

Validating the backend in icingaweb2->Configuration->Resources->icinga_ido seems OK

Validation Log

Connection to icinga2 as icingaweb on localhost: successful
have_ssl: DISABLED
protocol_version: 10
version: 5.5.60-0ubuntu0.14.04.1
version_compile_os: debian-linux-gnu

Debug log is showing nothing usefull

grep -v " debug/" /var/log/icinga2/debug.log  | grep -v "notice/Process" | grep -v "notice/Dependency" | grep -v " not in check period" | grep -v "Checkable: State Change"

[2018-07-23 10:35:12 +0200] information/ExternalCommandListener: 'command' started.
[2018-07-23 10:35:12 +0200] information/CheckerComponent: 'checker' started.
[2018-07-23 10:35:12 +0200] information/PerfdataWriter: 'perfdata' started.
[2018-07-23 10:35:12 +0200] information/DbConnection: 'ido-mysql' started.
[2018-07-23 10:35:12 +0200] information/NotificationComponent: 'notification' started.
[2018-07-23 10:35:12 +0200] information/ConfigItem: Activated all objects.
[2018-07-23 10:35:12 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonCommand::Run'
[2018-07-23 10:35:12 +0200] information/DbConnection: Resuming IDO connection: ido-mysql
[2018-07-23 10:35:12 +0200] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2018-07-23 10:35:12 +0200] notice/WorkQueue: Spawning WorkQueue threads for 'IdoMysqlConnection, ido-mysql'
[2018-07-23 10:35:12 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.0275151%
[2018-07-23 10:35:27 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.332661%
[2018-07-23 10:35:42 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.28743%
[...]

Please let me know how to debug this further, and I will be happy to help with more data.

Cheers and thanks, Tobias

dnsmichi commented 6 years ago

That's highly likely a duplicate of #6445.

TM08353 commented 6 years ago

Thank you. Confirmed. Missed that because it did not mention the DB.

Starting manually in a screen session without the "-d" after "deamon" is running as expected. Will use that as a workaround.

Icinga / icinga2