Icinga memory usage is increasing continuously.

samta93 commented 6 years ago

Expected Behavior

Memory usage should not be keep inscreasing.

Current Behavior

When adding more 1k host objects the memory usage for icinga process is keep increasing.

Possible Solution

Steps to Reproduce (for bugs)

Create 1000 host objects.

Context

We are creating 1k host objects, we see that there is api call to icinga api in the logs ([2018-04-12 04:13:52 -0700] information/HttpServerConnection: Request: GET /v1/objects/hosts (from [127.0.0.1]:34770, user: admin)) and right after each call memory usage is increasing.

This might be happening because after each api calls, we see a thread being spawned which never dies, ps -m pidof icinga2 | wc -l | xargs

Your Environment

Version used (icinga2 --version): (version: r2.5.4-1)
Operating System and version: Platform: Ubuntu Platform version: 14.04.5 LTS, Trusty Tahr Kernel: Linux Kernel version: 3.13.0-141-generic Architecture: x86_64
Enabled features (icinga2 feature list): Disabled features: compatlog gelf graphite influxdb livestatus opentsdb statusdata syslog Enabled features: api checker command debuglog ido-mysql mainlog notification perfdata
Icinga Web 2 version and modules (System - About):
Config validation (icinga2 daemon -C): information/cli: Icinga application loader (version: r2.5.4-1) information/cli: Loading configuration file(s). information/ConfigItem: Committing config item(s). information/ApiListener: My API identity: centralmsvm.englab.juniper.net

warning/ApplyRule: Apply rule 'disable-host-service-notifications' (in /etc/icinga2/conf.d/dependency.conf: 1:0-1:63) for type 'Dependency' does not match anywhere! warning/ApplyRule: Apply rule 'disable-host-service-checks' (in /etc/icinga2/conf.d/dependency.conf: 6:1-6:57) for type 'Dependency' does not match anywhere! warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere! warning/ApplyRule: Apply rule 'sendnotify' (in /etc/icinga2/conf.d/notifications.conf: 11:1-11:39) for type 'Notification' does not match anywhere! warning/ApplyRule: Apply rule 'sendnotify' (in /etc/icinga2/conf.d/notifications.conf: 17:1-17:42) for type 'Notification' does not match anywhere! warning/ApplyRule: Apply rule 'sendnotify_ne' (in /etc/icinga2/conf.d/notifications.conf: 23:1-23:42) for type 'Notification' does not match anywhere! warning/ApplyRule: Apply rule 'sendnotify_ne' (in /etc/icinga2/conf.d/notifications.conf: 29:1-29:45) for type 'Notification' does not match anywhere! warning/ApplyRule: Apply rule 'autoheal-sendnotify' (in /etc/icinga2/conf.d/notifications.conf: 50:1-50:48) for type 'Notification' does not match anywhere! warning/ApplyRule: Apply rule 'autoheal-sendnotify' (in /etc/icinga2/conf.d/notifications.conf: 56:1-56:51) for type 'Notification' does not match anywhere! warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/conf.d/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere! information/ConfigItem: Instantiated 2 ApiUsers. information/ConfigItem: Instantiated 1 ApiListener. information/ConfigItem: Instantiated 1 Zone. information/ConfigItem: Instantiated 2 FileLoggers. information/ConfigItem: Instantiated 1 Endpoint. information/ConfigItem: Instantiated 10 NotificationCommands. information/ConfigItem: Instantiated 23832 Notifications. information/ConfigItem: Instantiated 201 CheckCommands. information/ConfigItem: Instantiated 2754 Hosts. information/ConfigItem: Instantiated 1 IcingaApplication. information/ConfigItem: Instantiated 2784 HostGroups. information/ConfigItem: Instantiated 1 EventCommand. information/ConfigItem: Instantiated 1 UserGroup. information/ConfigItem: Instantiated 1 User. information/ConfigItem: Instantiated 21078 Services. information/ConfigItem: Instantiated 3 TimePeriods. information/ConfigItem: Instantiated 3 ServiceGroups. information/ConfigItem: Instantiated 1 CheckerComponent. information/ConfigItem: Instantiated 1 ExternalCommandListener. information/ConfigItem: Instantiated 1 IdoMysqlConnection. information/ConfigItem: Instantiated 1 NotificationComponent. information/ConfigItem: Instantiated 1 PerfdataWriter. information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' information/cli: Finished validating the configuration file(s)

If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

dnsmichi commented 6 years ago

v2.5.x is not supported anymore. In terms of the problem, this is a duplicate of #5148.

samta93 commented 6 years ago

Hi @dnsmichi

Is this resolved #5148 with latest icinga, I have upgraded to r2.8.2-1 and still saw the same issue.

dnsmichi commented 6 years ago

The linked issue is still open and under investigation.

samta93 commented 6 years ago

Thanks @dnsmichi I will wait for this issue to get resolved. Please let me know if need any help with test and verification.

Icinga / icinga2