Alignak-monitoring / alignak

Monitoring tool, highly flexible and new standard oriented
https://alignak-monitoring.github.io
GNU Affero General Public License v3.0
86 stars 19 forks source link

Daemons interface timeout, less than expected 3 seconds #1006

Open mohierf opened 6 years ago

mohierf commented 6 years ago

Daemons communication timeout lokks strange:

[2018-03-15 04:58:08] INFO: [arbiter-master.alignak.daemon] [arbiter-master] starting main loop: 1521040150.91
[2018-03-15 04:58:43] WARNING: [arbiter-master.alignak.daemon] The arbiter arbiter-master loop exceeded the maximum expected loop duration: 1.00. The last loop needed 1.94 seconds t
o execute. You should try to reduce the load on this arbiter.
[2018-03-15 04:58:57] WARNING: [arbiter-master.alignak.objects.satellitelink] Add failed attempt for broker-master (1/3) - Connection timeout with 'get_conf': Request timeout (3 sec
onds) for http://127.0.0.1:7772/get_managed_configurations
[2018-03-15 04:58:57] WARNING: [arbiter-master.alignak.daemon] The arbiter arbiter-master loop exceeded the maximum expected loop duration: 1.00. The last loop needed 3.19 seconds t
o execute. You should try to reduce the load on this arbiter.
[2018-03-15 04:58:57] ERROR: [arbiter-master.alignak.objects.satellitelink] The broker broker-master is not reachable
[2018-03-15 04:58:57] WARNING: [arbiter-master.alignak.objects.satellitelink] Add failed attempt for broker-master (2/3) - Connection timeout with 'update_infos': Satellite link err
or: The broker broker-master is not reachable
[2018-03-15 04:58:58] ERROR: [arbiter-master.alignak.objects.satellitelink] The broker broker-master is not reachable
[2018-03-15 04:58:58] WARNING: [arbiter-master.alignak.objects.satellitelink] Add failed attempt for broker-master (3/3) - Connection timeout with 'update_infos': Satellite link err
or: The broker broker-master is not reachable
[2018-03-15 04:58:58] WARNING: [arbiter-master.alignak.objects.satellitelink] Set broker-master as dead, too much failed attempts (3), last problem is: Connection timeout with 'upda
te_infos': Satellite link error: The broker broker-master is not reachable
[2018-03-15 04:58:58] WARNING: [arbiter-master.alignak.objects.satellitelink] Setting the satellite broker-master as dead :(
[2018-03-15 04:58:59] ERROR: [arbiter-master.alignak.objects.satellitelink] The connection is not created for broker-master

3 seconds timeout expire during the same second :(

mohierf commented 6 years ago

Another example:

[2018-03-15 07:57:05] WARNING: [arbiter-master.alignak.objects.satellitelink] Add failed attempt for broker-master (1/3) - Connection timeout with 'get_conf': Request timeout (3 seconds) for http://127.0.0.1:7772/get_managed_configurations
[2018-03-15 07:57:05] WARNING: [arbiter-master.alignak.daemon] The arbiter arbiter-master loop exceeded the maximum expected loop duration: 1.00. The last loop needed 3.19 seconds to execute. You should try to reduce the load on this arbiter.
[2018-03-15 07:57:05] ERROR: [arbiter-master.alignak.objects.satellitelink] The broker broker-master is not reachable
[2018-03-15 07:57:05] WARNING: [arbiter-master.alignak.objects.satellitelink] Add failed attempt for broker-master (2/3) - Connection timeout with 'update_infos': Satellite link error: The broker broker-master is not reachable
[2018-03-15 07:57:06] ERROR: [arbiter-master.alignak.objects.satellitelink] The broker broker-master is not reachable
[2018-03-15 07:57:06] WARNING: [arbiter-master.alignak.objects.satellitelink] Add failed attempt for broker-master (3/3) - Connection timeout with 'update_infos': Satellite link error: The broker broker-master is not reachable
[2018-03-15 07:57:06] WARNING: [arbiter-master.alignak.objects.satellitelink] Set broker-master as dead, too much failed attempts (3), last problem is: Connection timeout with 'update_infos': Satellite link error: The broker broker-master is not reachable
[2018-03-15 07:57:06] WARNING: [arbiter-master.alignak.objects.satellitelink] Setting the satellite broker-master as dead :(
[2018-03-15 07:57:07] INFO: [arbiter-master.alignak.objects.satellitelink]   get the running identifier for broker broker-master.
[2018-03-15 07:57:12] INFO: [broker-master.alignak.daemon] Daemon broker-master is living: loop #1 ;)
[2018-03-15 07:57:12] WARNING: [broker-master.alignak.daemon] The broker broker-master loop exceeded the maximum expected loop duration: 1.00. The last loop needed 72.99 seconds to execute. You should try to reduce the load on this broker.
[2018-03-15 07:57:13] INFO: [broker-master.alignak.module.backend_broker] Got a new configuration, reloading objects...