search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2.03k stars 579 forks source link

[dev.icinga.com #8880] Too many established API-connections on one taget #2833

Closed icinga-migration closed 8 years ago

icinga-migration commented 9 years ago

This issue has been migrated from Redmine: https://dev.icinga.com/issues/8880

Created by nuts on 2015-03-27 11:36:43 +00:00

Assignee: (none) Status: Closed (closed on 2016-02-25 00:21:36 +00:00) Target Version: (none) Last Update: 2016-02-25 00:21:36 +00:00 (in Redmine)

Icinga Version: 2.3.2
Backport?: Not yet backported
Include in Changelog: 1

Icinga2 Master instance opens to much API connections to one target. I have noticed that problem because the client can't handle the load. This is a part of the client log:

[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: PID 25483 ('/usr/sbin/dhcpstat' '71') terminated with exit code 0
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: PID 25492 ('/usr/sbin/dhcpstat' '71') terminated with exit code 0
[2015-03-27 01:07:40 +0100] notice/ApiListener: Sending message to 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/ApiClient: Received 'event::ExecuteCommand' message from 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: Running command '/usr/lib/nagios/plugins/check_procs' '-c' '10' '-s' 'Z' '-w' '5': PID 25690
[2015-03-27 01:07:40 +0100] notice/ApiClient: Received 'event::ExecuteCommand' message from 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: Running command '/usr/lib/nagios/plugins/check_procs' '-c' '10' '-s' 'Z' '-w' '5': PID 25692
[2015-03-27 01:07:40 +0100] notice/ApiClient: Received 'event::ExecuteCommand' message from 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: Running command '/usr/lib/nagios/plugins/check_procs' '-c' '10' '-s' 'Z' '-w' '5': PID 25693
[2015-03-27 01:07:40 +0100] notice/ApiClient: Received 'event::ExecuteCommand' message from 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: Running command '/usr/lib/nagios/plugins/check_procs' '-c' '10' '-s' 'Z' '-w' '5': PID 25694
[2015-03-27 01:07:40 +0100] notice/ApiClient: Received 'event::ExecuteCommand' message from 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: Running command '/usr/lib/nagios/plugins/check_procs' '-c' '10' '-s' 'Z' '-w' '5': PID 25695
[2015-03-27 01:07:40 +0100] notice/ApiClient: Received 'event::ExecuteCommand' message from 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: Running command '/usr/lib/nagios/plugins/check_procs' '-c' '10' '-s' 'Z' '-w' '5': PID 25696
[2015-03-27 01:07:40 +0100] notice/ApiClient: Received 'event::ExecuteCommand' message from 'bigsister.fem.tu-ilmenau.de'
[2015-03-27 01:07:40 +0100] notice/Process: Running command '/usr/lib/nagios/plugins/check_procs' '-c' '10' '-s' 'Z' '-w' '5': PID 25703

--> Seven times check_procs and eight tcp connections to :5665 in this moment.

The master and the client configured in remote check execution mode. On the master are two host objects für the client because a HA setup for our dhcp-sercive. Both objects terminates on the same client with different IPs.

Server-config:

object Endpoint "admindb2-dhcp-01.fem.tu-ilmenau.de" {
  host = "admindb2-dhcp-01.fem.tu-ilmenau.de"
}

object Zone "admindb2-dhcp-01.fem.tu-ilmenau.de" {
        endpoints = [ "admindb2-dhcp-01.fem.tu-ilmenau.de" ]
        parent = "master"
}

object Host "admindb2-dhcp-01" {
  import "generic-host"
  address = "10.42.49.215"
#  address6 = "::1"
  vars.remote_client = "admindb2-dhcp-01.fem.tu-ilmenau.de"
# remote execution checks (e.g. check_procs)
# authenticated and registered on masters api
}

HA-IP config:

object Endpoint "admindb2-dhcp.net.fem.tu-ilmenau.de" {
#  host = "192.168.33.20"
  host = "admindb2-dhcp.net.fem.tu-ilmenau.de"
}

object Zone "admindb2-dhcp.net.fem.tu-ilmenau.de" {
        endpoints = [ "admindb2-dhcp.net.fem.tu-ilmenau.de" ]
        parent = "master"
}

object Host "admindb2-dhcp" {
  import "generic-host"
  address = "10.42.1.1"
  vars.nrpe = "true"
# _no_ remote execution checks, only nrpe
# not authenticated and not registered on masters api
}

Within a few minutes more and more tcp connections established on the client (see above:8). The interesting point is that the mass connections (7x) come from the "unregistered" (admindb2-dhcp) object. The "registered" object (admindb2-dhcp-01) works fine. Only one tcp connection only one check whithin the checkintervall.

The setup is running atm with an inactive icinga2 instance on the client. So its easy for me providing more useful informations if they are needed.

icinga-migration commented 9 years ago

Updated by mfriedrich on 2015-04-17 11:42:56 +00:00

icinga-migration commented 9 years ago

Updated by tobiasvdk on 2015-04-20 20:02:33 +00:00

Add formatting.

icinga-migration commented 9 years ago

Updated by mfriedrich on 2015-07-02 18:46:20 +00:00

icinga-migration commented 8 years ago

Updated by mfriedrich on 2016-02-25 00:21:37 +00:00

This has been fixed in the recent release.