search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2.01k stars 577 forks source link

Icinga2 CPU utilization went up after updating to 2.13.5 for both Master and Satellite server (EC2 instance on AWS) #9531

Open Prathameshhankare opened 2 years ago

Prathameshhankare commented 2 years ago

Describe the bug

After upgrading the Icinga2 master and the satellite server the CPU utilization started to raise. Before the upgrade, the CPU utilization was below 20% avg and now after the upgrade, the CPU utilization is at 40% avg.

I tried looking for logs and I am unable to find the exact reason for the sudden raise in CPU utilization.

Your Environment

` icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.5-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

System information: Platform: CentOS Linux Platform version: 7 (Core) Kernel: Linux Kernel version: 3.10.0-1160.76.1.el7.x86_64 Architecture: x86_64

Build information: Compiler: GNU 11.2.1 Build host: runner-hh8q3bz2-project-322-concurrent-0 OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017

Application information:

General paths: Config directory: /etc/icinga2 Data directory: /var/lib/icinga2 Log directory: /var/log/icinga2 Cache directory: /var/cache/icinga2 Spool directory: /var/spool/icinga2 Run directory: /run/icinga2

Old paths (deprecated): Installation root: /usr Sysconf directory: /etc Run directory (base): /run Local state directory: /var

Internal paths: Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /run/icinga2/icinga2.pid icinga2 feature list Disabled features: compatlog debuglog elasticsearch gelf icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog Enabled features: api checker command graphite ido-mysql mainlog notification icinga2 daemon -C [2022-09-29 09:01:07 +0000] information/cli: Icinga application loader (version: r2.13.5-1) [2022-09-29 09:01:07 +0000] information/cli: Loading configuration file(s). [2022-09-29 09:01:07 +0000] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/IC-Satellite-Server' for unknown zone 'IC-Satellite-Server'. [2022-09-29 09:01:07 +0000] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/IT Satellite' for unknown zone 'IT Satellite'. [2022-09-29 09:01:07 +0000] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/ITALY' for unknown zone 'ITALY'. [2022-09-29 09:01:07 +0000] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/ITSUPSAT01-Satellite' for unknown zone 'ITSUPSAT01-Satellite'. [2022-09-29 09:01:07 +0000] information/ConfigItem: Committing config item(s). [2022-09-29 09:01:07 +0000] information/ApiListener: My API identity: EUAWSICN01 [2022-09-29 09:01:17 +0000] information/WorkQueue: #5 (DaemonUtility::LoadConfigFiles) items: 0, rate: 4.7/s (282/min 282/5min 282/15min); [2022-09-29 09:01:17 +0000] information/WorkQueue: #6 (GraphiteWriter, graphite) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2022-09-29 09:01:17 +0000] information/WorkQueue: #8 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2022-09-29 09:01:17 +0000] information/WorkQueue: #9 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2022-09-29 09:01:26 +0000] warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/conf.d/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere! [2022-09-29 09:01:26 +0000] warning/ApplyRule: Apply rule 'ping4' (in /etc/icinga2/conf.d/services.conf: 27:1-27:21) for type 'Service' does not match anywhere! [2022-09-29 09:01:26 +0000] warning/ApplyRule: Apply rule 'swap' (in /etc/icinga2/conf.d/services.conf: 105:1-105:20) for type 'Service' does not match anywhere! [2022-09-29 09:01:26 +0000] warning/ApplyRule: Apply rule 'Disk_Windows' (in /var/lib/icinga2/api/packages/director/f1f0f134-fb03-4e57-a651-c5ee155857b1/zones.d/director-global/service_apply.conf: 41:1-41:28) for type 'Service' does not match anywhere! [2022-09-29 09:01:26 +0000] warning/ApplyRule: Apply rule 'Memcached Service' (in /var/lib/icinga2/api/packages/director/f1f0f134-fb03-4e57-a651-c5ee155857b1/zones.d/director-global/service_apply.conf: 105:1-105:33) for type 'Service' does not match anywhere! [2022-09-29 09:01:26 +0000] warning/ApplyRule: Apply rule 'MySQL Service' (in /var/lib/icinga2/api/packages/director/f1f0f134-fb03-4e57-a651-c5ee155857b1/zones.d/director-global/service_apply.conf: 149:1-149:29) for type 'Service' does not match anywhere! [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 GraphiteWriter. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 NotificationComponent. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 CheckerComponent. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 5 UserGroups. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 7 TimePeriods. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 9 Users. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 20836 Services. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 27 ServiceGroups. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 793 Zones. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 7 NotificationCommands. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 10587 Notifications. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 2044 Hosts. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 IcingaApplication. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 103 HostGroups. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 296 Comments. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 Downtime. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 790 Endpoints. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 FileLogger. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 4 ApiUsers. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 271 CheckCommands. [2022-09-29 09:01:26 +0000] information/ConfigItem: Instantiated 1 ApiListener. [2022-09-29 09:01:26 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2022-09-29 09:01:26 +0000] information/cli: Finished validating the configuration file(s). `

julianbrost commented 2 years ago

What was the version before the update? 2.13.4? Or something older?

Prathameshhankare commented 2 years ago

Yes, it was 2.13.4 only.

julianbrost commented 2 years ago

That's strange as there's no change between these versions that should explain this. Given the features you use, the only change that should really affect you would be #9451. Do you have any graphs of the CPU usage available? Does it show any patterns like increased usage every 5 minutes?

Prathameshhankare commented 2 years ago

As soon as we start using Icingaweb2 the CPU starts utilizing a lot and once the shift is ended, the CPU utilization comes down.

Is there a way to downgrade the Icinga2? image

julianbrost commented 2 years ago

For downgrading, please refer to the documentation for your package manager, for CentOS that should be possible with yum downgrade.

But if the CPU usage correlates with the use of Icinga Web 2, have you updated this recently?

Prathameshhankare commented 2 years ago

Yes, I updated everything including icinga2, icingaweb2, Carbon & director as well.

Al2Klimov commented 1 year ago
  1. From/to which versions?
  2. The downgrade of which component fixes the problem?