search

sni / lmd

Livestatus Multitool Daemon - Create livestatus federation from multiple sources
https://labs.consol.de/omd/packages/lmd/
GNU General Public License v3.0
42 stars 31 forks source link

Thruk/LMD: The LMD process sometimes disappears without a trace #109

Closed steffen-poulsen closed 3 years ago

steffen-poulsen commented 3 years ago

Having the #108 issue explained (thanks! :-)), next up for us is another similar issue, where the LMD process appears to just vanish without a trace.

It is almost like the LMD process is being hard killed and then restarted by Thruk - except, no trace of this happening is left in the thruk.log file.

The pattern in the log file goes like this:

The last message being the ordinary LMD startup message.

The startup message happens just split seconds after the latest entry of the ordinary activity, so the LMD process appears to be restarted blazingly fast.

Any ideas what might be the cause of this pattern?

[root@dkrdswppthrukp01 ~]# ps -ef | grep lmd
apache   32493     1 99 02:38 ?        05:04:57 /usr/local/bin/lmd -pidfile /var/cache/thruk/lmd/pid -config /var/cache/thruk/lmd/lmd.ini -config /etc/thruk/thruk_local.d/lmd.ini
root     22203  4843  0 04:40 pts/2    00:00:00 grep lmd
[root@dkrdswppthrukp01 ~]# 

# cat /var/log/lmd.log | egrep '(02\:38\:|02\:37\:|02\:36\:)' | grep -v "request.go:313"
[2021-01-22 02:38:48.236][Debug][listener.go:75] incoming request from: @ to /var/cache/thruk/lmd/live.sock
[2021-01-22 02:38:48.236][Debug][request.go:291] request: GET services
[2021-01-22 02:38:48.338][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 98.803725ms, response size: 90 B
[2021-01-22 02:38:48.342][Debug][listener.go:75] incoming request from: @ to /var/cache/thruk/lmd/live.sock
[2021-01-22 02:38:48.342][Debug][request.go:291] request: GET hosts
[2021-01-22 02:38:48.791][Debug][main.go:880] command line arguments:
[2021-01-22 02:38:48.792][Debug][main.go:882] args: [
[2021-01-22 02:38:48.792][Debug][main.go:882] args:   "/usr/local/bin/lmd",
[2021-01-22 02:38:48.792][Debug][main.go:882] args:   "-pidfile",
[2021-01-22 02:38:48.792][Debug][main.go:882] args:   "/var/cache/thruk/lmd/pid",
[2021-01-22 02:38:48.792][Debug][main.go:882] args:   "-config",
[2021-01-22 02:38:48.792][Debug][main.go:882] args:   "/var/cache/thruk/lmd/lmd.ini",
[2021-01-22 02:38:48.792][Debug][main.go:882] args:   "-config",
...
[2021-01-22 02:38:48.795][Debug][main.go:889] conf:   "CompressionMinimumSize": 500,
[2021-01-22 02:38:48.795][Debug][main.go:889] conf:   "CompressionLevel": -1,
[2021-01-22 02:38:48.795][Debug][main.go:889] conf:   "MaxClockDelta": 0,
[2021-01-22 02:38:48.795][Debug][main.go:889] conf:   "UpdateOffset": 3
[2021-01-22 02:38:48.795][Debug][main.go:889] conf: }
[2021-01-22 02:38:48.796][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-22 02:38:48.797][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-22 02:38:48.797][Info][listener.go:305] listening for incoming queries on tcp :3333
[2021-01-22 02:38:48.797][Info][peer.go:309] [NCE-3] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [MED-2] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [APPL-1] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [APPL-ChinaHUB] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [Master] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [NCE-1] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [MED-1] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [ASP-1] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [MED-3] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [NAG99] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [NCE-2] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [NCE-4] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [CHI-1] starting connection
[2021-01-22 02:38:48.797][Info][peer.go:309] [AME-1] starting connection
[2021-01-22 02:38:48.799][Debug][datastoreset.go:93] [Master] updated table:          status - fetch:       0s - insert:
 0s - count:        1 - size:        0 kB
[2021-01-22 02:38:48.800][Debug][datastoreset.go:93] [AME-1] updated table:          status - fetch:      1ms - insert:
0s - count:        1 - size:        0 kB
[2021-01-22 02:38:48.800][Debug][peer.go:1059] [Master] LastQuery:
[2021-01-22 02:38:48.800][Debug][peer.go:1060] [Master] GET columns
[2021-01-22 02:38:48.800][Debug][peer.go:907] [Master] sending data/query failed: [Master] bad response code: 404 - Table 'col
umns' does not exist.
[2021-01-22 02:38:48.800][Debug][datastoreset.go:93] [NCE-3] updated table:          status - fetch:      1ms - insert:
0s - count:        1 - size:        0 kB
[2021-01-22 02:38:48.800][Debug][datastoreset.go:93] [NCE-1] updated table:          status - fetch:      2ms - insert:
0s - count:        1 - size:        0 kB

[root@dkrdswppthrukp01 ~]# ps -ef | grep lmd
apache    8139     1 99 09:04 ?        00:28:17 /usr/local/bin/lmd -pidfile /var/cache/thruk/lmd/pid -config /var/cache/thruk/lmd/lmd.ini -config /etc/thruk/thruk_local.d/lmd.ini
root     10055  4843  0 09:12 pts/2    00:00:00 grep lmd
[root@dkrdswppthrukp01 ~]#

[2021-01-22 09:04:35.131][Debug][datastoreset.go:300] [ASP-1] updated table:           hosts - fetch:    111ms - insert:       0s - count:       12 - size:        6 kB
[2021-01-22 09:04:35.143][Debug][datastoreset.go:300] [ASP-1] updated table:        services - fetch:     10ms - insert:       0s - count:       15 - size:        5 kB
[2021-01-22 09:04:35.151][Debug][datastoreset.go:199] [ASP-1] delta update complete in: 134.949993ms
[2021-01-22 09:04:35.244][Debug][datastoreset.go:300] [NCE-4] updated table:           hosts - fetch:     32ms - insert:    276ms - count:        1 - size:        0 kB
[2021-01-22 09:04:35.795][Debug][main.go:880] command line arguments:

[root@dkrdswppthrukp01 ~]# ps -ef | grep lmd
apache   22269     1 99 10:06 ?        11:09:18 /usr/local/bin/lmd -pidfile /var/cache/thruk/lmd/pid -config /var/cache/thruk/lmd/lmd.ini -config /etc/thruk/thruk_local.d/lmd.ini
root     15810  4843  0 12:06 pts/2    00:00:00 grep lmd
[root@dkrdswppthrukp01 ~]#

[2021-01-22 10:06:57.340][Debug][request.go:313] request: ResponseHeader: fixed16
[2021-01-22 10:06:57.415][Debug][rawresultset.go:39] sorting result took 52.417µs
[2021-01-22 10:06:57.418][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 77.826943ms, response size: 281.2KiB
[2021-01-22 10:06:57.493][Debug][datastoreset.go:708] [AME-1] updated table:          status - fetch:      1ms - insert:       0s - count:        1 - size:        0 kB
[2021-01-22 10:06:58.207][Debug][main.go:880] command line arguments:

[root@dkrdswppthrukp01 ~]# ps -ef | grep lmd
apache   16223     1 99 12:08 ?        00:01:00 /usr/local/bin/lmd -pidfile /var/cache/thruk/lmd/pid -config /var/cache/thruk/lmd/lmd.ini -config /etc/thruk/thruk_local.d/lmd.ini
root     16635  4843  0 12:09 pts/2    00:00:00 grep lmd
[root@dkrdswppthrukp01 ~]# 

[2021-01-22 12:08:51.981][Debug][datastoreset.go:300] [NCE-4] updated table:        services - fetch:     25ms - insert:       0s - count:        1 - size:        0 kB
[2021-01-22 12:08:51.984][Debug][datastoreset.go:300] [NCE-4] updated table:        services - fetch:     33ms - insert:       0s - count:        1 - size:        0 kB
[2021-01-22 12:08:51.984][Debug][datastoreset.go:300] [NCE-4] updated table:        services - fetch:     23ms - insert:       0s - count:        1 - size:        0 kB
[2021-01-22 12:08:51.986][Debug][datastoreset.go:300] [NCE-4] updated table:        services - fetch:     35ms - insert:       0s - count:        1 - size:        0 kB
[2021-01-22 12:08:52.294][Debug][main.go:880] command line arguments:

[root@dkrdswppthrukp01 ~]# ps -ef | grep lmd
apache   17230     1 99 12:11 ?        00:23:47 /usr/local/bin/lmd -pidfile /var/cache/thruk/lmd/pid -config /var/cache/thruk/lmd/lmd.ini -config /etc/thruk/thruk_local.d/lmd.ini
root     18440  4843  0 12:18 pts/2    00:00:00 grep lmd
[root@dkrdswppthrukp01 ~]# 

[2021-01-22 12:11:29.607][Debug][request.go:313] request: Columns: can_submit_commands alias email peer_key
[2021-01-22 12:11:29.607][Debug][request.go:313] request: Filter: name = yaylu
[2021-01-22 12:11:29.607][Debug][request.go:313] request: OutputFormat: json
[2021-01-22 12:11:29.607][Debug][request.go:313] request: ResponseHeader: fixed16
[2021-01-22 12:11:29.929][Debug][main.go:880] command line arguments:
[root@dkrdswppthrukp01 ~]#

Looking at this from the Thruk side of things, it doesn't look like Thruk is the cause of this.

The log doesn't mention that Thruk is killing the LMD process, or even restarting it.

[2021/01/22 12:05:54][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 12:06:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 12:07:20][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 12:07:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 12:08:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 12:08:53][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] internal lmd error - did not get a valid response for at least any site at /usr/share/thruk/lib/Thruk/Backend/Manager.pm line 1880.
[2021/01/22 12:19:25][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0

The Thruk log in more detail.

From losing connection to LMD, to the LMD having been restarted, to all backends up again.

# tail -20000 /var/log/thruk/thruk.log | egrep -v '^\||^\+' 
[2021/01/22 12:08:31][dkrdswppthrukp01.vestasext.net][WARN] params:  {'active_downtimes' => 'on','ahas' => '','broadcast_notification' => '','childoptions' => '0','cmd_mod' => 2,'cmd_typ' => 7,'com_author' => 'Leanne Mae Dayao','com_data' => '','com_id' => 0,'down_id' => 0,'end_time' => '2021-01-22 14:05:00','expir...
[2021/01/22 12:08:31][dkrdswppthrukp01.vestasext.net][WARN] user:    laeda
[2021/01/22 12:08:31][dkrdswppthrukp01.vestasext.net][WARN] address: 10.219.226.231
[2021/01/22 12:08:31][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:08:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0                                                                                        [2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 102.1s to load.
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/cmd.cgi
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] params:  {'active_downtimes' => 'on','ahas' => '','broadcast_notification' => '','childoptions' => '0','cmd_mod' => 2,'cmd_typ' => 7,'com_author' => 'John Carl Marzan','com_data' => '','com_id' => 0,'down_id' => 0,'end_time' => '2021-01-22 14:06:00','expir...
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] user:    jalmr
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] address: 10.219.224.109
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 75.3s to load.
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/cmd.cgi
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] params:  {'active_downtimes' => 'on','ahas' => '','broadcast_notification' => '','childoptions' => '0','cmd_mod' => 2,'cmd_typ' => 7,'com_author' => 'Leanne Mae Dayao','com_data' => '','com_id' => 0,'down_id' => 0,'end_time' => '2021-01-22 14:07:00','expir...
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] user:    laeda
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] address: 10.219.226.231
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 120.8s to load.
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/cmd.cgi
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] params:  {'active_downtimes' => 'on','ahas' => '','broadcast_notification' => '','childoptions' => '0','cmd_mod' => 2,'cmd_typ' => 7,'com_author' => 'Leanne Mae Dayao','com_data' => '','com_id' => 0,'down_id' => 0,'end_time' => '2021-01-22 14:05:00','expir...
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] user:    laeda
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] address: 10.219.226.231
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?hostgroup=all&style=summary&_=1611313729292
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611313729292','hostgroup' => 'all','style' => 'summary'};
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] user:    jfgda
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.10.61.22
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] CHI-1: peer is down: connecting... (172.26.66.63:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NAG99: peer is down: connecting... (172.26.66.12:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] APPL-1: peer is down: connecting... (10.0.53.142:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] APPL-ChinaHUB: peer is down: connecting... (10.78.8.134:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] did not get a valid response for at least any site at /usr/share/thruk/lib/Thruk/Backend/Manager.pm line 1880.
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?backends=51656&backends=4120b&backends=a103e&backends=53dec&backends=d4c78&backends=bc29d&backends=51d28&backends=bde22&backends=90ba2&backends=5f10e&backends=a690a&backends=4120b&backends=a103e&backends=53dec&backends=d4c78&backends=bc29d&backends=51d28&backends=bde22&backends=90ba2&backends=5f10e&backends=a690a&dfl_s0_hostprops=262186&dfl_s0_hoststatustypes=2&dfl_s0_op==&dfl_s0_op=~&dfl_s0_op=!=&dfl_s0_serviceprops=262186&dfl_s0_servicestatustypes=16&dfl_s0_type=custom+variable&dfl_s0_type=host&dfl_s0_type=service&dfl_s0_val_pre=OPRSTATUS&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_value=InService&dfl_s0_value=PPC-MAIN&dfl_s0_value=ppc_time&dfl_s1_hostprops=262186&dfl_s1_hoststatustypes=2&dfl_s1_op=~&dfl_s1_op=!=&dfl_s1_op=!=&dfl_s1_op=!=&dfl_s1_op=!~&dfl_s1_op=!=&dfl_s1_op=!=&dfl_s1_op=!=&dfl_s1_op=!=&dfl_s1_op=!=&dfl_s1_op==&dfl_s1_op=!~&dfl_s1_serviceprops=262186&dfl_s1_servicestatustypes=16&dfl_s1_type=host&dfl_s1_type=service&dfl_s1_type=service&dfl_s1_type=service&dfl_s1_type=plugin+output&dfl_s1_type=service&dfl_s1_type=service&dfl_s1_type=service&dfl_s1_type=service&dfl_s1_type=service&dfl_s1_type=custom+variable&dfl_s1_type=service&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=OPRSTATUS&dfl_s1_val_pre=&dfl_s1_value=PPC-Log&dfl_s1_value=wsus_updates&dfl_s1_value=eventlog_Windows_System&dfl_s1_value=eventlog_Puppet_Agent&dfl_s1_value=unlock_puppet&dfl_s1_value=task_unlock_puppet_agent&dfl_s1_value=process_mem_3rd_MongoDB&dfl_s1_value=ntp_timesync&dfl_s1_value=Connectivity&dfl_s1_value=connectivity_RDP&dfl_s1_value=InService&dfl_s1_value=disk_c&scrollTo=0
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'backends' => ['51656','4120b','a103e','53dec','d4c78','bc29d','51d28','bde22','90ba2','5f10e','a690a','4120b','a103e','53dec','d4c78','bc29d','51d28','bde22','90ba2','5f10e','a690a'],'dfl_s0_hostprops' => '262186','dfl_s0_hoststatustypes' => '2'...
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] user:    laeda
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.226.231
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] APPL-1: peer is down: connecting... (10.0.53.142:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?host=DE-Wildenberg-PPC-MAIN&scrollTo=0
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'host' => 'DE-Wildenberg-PPC-MAIN','scrollTo' => '0'};
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] user:    laeda
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.226.231
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:52][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:53][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0                                                                                        [2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 48.9s to load.
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/cmd.cgi
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 62.6s to load.
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/cmd.cgi
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] params:  {'active_downtimes' => 'on','ahas' => '','broadcast_notification' => '','childoptions' => '0','cmd_mod' => 2,'cmd_typ' => 34,'com_author' => 'John Carl Marzan','com_data' => '309979205','com_id' => 0,'down_id' => 0,'end_time' => '2021-01-22 14:06:...
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] user:    jalmr
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] address: 10.219.224.109
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] params:  {'active_downtimes' => 'on','ahas' => '','broadcast_notification' => '','childoptions' => '0','cmd_mod' => 2,'cmd_typ' => 7,'com_author' => 'Leanne Mae Dayao','com_data' => '','com_id' => 0,'down_id' => 0,'end_time' => '2021-01-22 14:07:00','expir...
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] user:    laeda
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] address: 10.219.226.231
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 55.6s to load.
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/cmd.cgi
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] params:  {'active_downtimes' => 'on','ahas' => '','broadcast_notification' => '','childoptions' => '0','cmd_mod' => 2,'cmd_typ' => 7,'com_author' => 'Leanne Mae Dayao','com_data' => '','com_id' => 0,'down_id' => 0,'end_time' => '2021-01-22 14:07:00','expir...
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] user:    laeda
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] address: 10.219.226.231
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?host=DE-SchneebergerhofWindhuebel-PPC-MAIN&scrollTo=0
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'host' => 'DE-SchneebergerhofWindhuebel-PPC-MAIN','scrollTo' => '0'};
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] user:    laeda
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.226.231
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:54][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?nav=&hidesearch=0&hidetop=&dfl_columns=host_name,description,state,last_check,duration,plugin_output,cust_OPRSTATUS,host_address,cust_HOSTNETWORK,cust_CUSTCATEGORY&style=detail&dfl_s0_type=host&dfl_s0_type=service&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_op=~&dfl_s0_op==&dfl_s0_value=-WAN1&dfl_s0_value=ntp_timesync&sortoption=9&sorttype=2&reload_nav=1&scrollTo=400&_=1611313733465
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611313733465','dfl_columns' => 'host_name,description,state,last_check,duration,plugin_output,cust_OPRSTATUS,host_address,cust_HOSTNETWORK,cust_CUSTCATEGORY','dfl_s0_op' => ['~','='],'dfl_s0_type' => ['host','service'],'dfl_s0_val_pre' =...
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] user:    chj
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.10.48.68
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?host=DE-Schluechtern-PPC-MAIN&scrollTo=0
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'host' => 'DE-Schluechtern-PPC-MAIN','scrollTo' => '0'};
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] user:    laeda
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.226.231
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:55][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/parts.cgi?part=_host_downtimes&host=DE-TennenbronnVOG-VOC1-SRV&backend=d4c78
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'backend' => 'd4c78','host' => 'DE-TennenbronnVOG-VOC1-SRV','part' => '_host_downtimes'};
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] user:    jalmr
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.224.109
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?host=TH-Romklao-PPC-MAIN
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'host' => 'TH-Romklao-PPC-MAIN'};
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] user:    laeda
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.226.231
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:56][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?nav=&hidesearch=0&hidetop=&dfl_columns=host_name,description,state,last_check,duration,plugin_output,cust_OPRSTATUS,host_address,cust_HOSTNETWORK,cust_CUSTCATEGORY&style=detail&update.x=11&update.y=6&dfl_s0_hoststatustype=2&dfl_s0_hoststatustype=4&dfl_s0_hoststatustype=8&dfl_s0_hoststatustype=1&dfl_s0_type=custom variable&dfl_s0_type=service&dfl_s0_type=custom variable&dfl_s0_type=custom variable&dfl_s0_val_pre=OPRSTATUS&dfl_s0_val_pre=&dfl_s0_val_pre=REGION&dfl_s0_val_pre=COUNTRY&dfl_s0_op==&dfl_s0_op==&dfl_s0_op==&dfl_s0_op==&dfl_s0_value=InService&dfl_s0_value=wsus_updates&dfl_s0_value=NCE&dfl_s0_value=RU&_=1611313736658
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611313736658','dfl_columns' => 'host_name,description,state,last_check,duration,plugin_output,cust_OPRSTATUS,host_address,cust_HOSTNETWORK,cust_CUSTCATEGORY','dfl_s0_hoststatustype' => ['2','4','8','1'],'dfl_s0_op' => ['=','=','=','='],'...
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] user:    chj
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.10.48.68
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:57][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?host=DE-Rossau-PPC-MAIN
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'host' => 'DE-Rossau-PPC-MAIN'};
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] user:    laeda
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.226.231
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:08:59][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/panorama.cgi?task=dashboard_save_states
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'current_tab' => 'tabpan-tab_96','nr' => 'tabpan-tab_96','states' => '{"tabpan-tab_96_panlet_24":{"state":3},"tabpan-tab_96_panlet_12":{"state":3},"tabpan-tab_96_panlet_14":{},"tabpan-tab_96_panlet_2":{"state":3},"tabpan-tab_96_panlet_23":{},"tab...
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] user:    laeda
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.226.231
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:09:00][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/panorama.cgi?task=status
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'backends' => ['a103e','5f10e','53dec','bde22','4120b','51656','bc29d','51d28','d4c78','a690a'],'current_tab' => 'tabpan-tab_269','reschedule' => '','state_type' => 'hard','task' => 'status','types' => '{"filter":{"[\\"on\\",null,\\"[{\\\\\\"host...
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] user:    chj
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.10.48.68
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/extinfo.cgi?type=2&host=DK-Oesterild-WAN1&service=check_cdp_neighbors&backend=a690a&scrollTo=1483&_=1611313743852
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611313743852','backend' => 'a690a','host' => 'DK-Oesterild-WAN1','scrollTo' => '1483','service' => 'check_cdp_neighbors','type' => '2'};
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] user:    nnit-espi
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.10.40.196
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:09:05][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:09:08][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:09:08][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/extinfo.cgi?type=2&host=DK-HED42-MK5-Testlab-VOB1-SRV1&service=mssql_database-file-dbcc-shrinks&backend=a690a&_=1611313746848
[2021/01/22 12:09:08][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611313746848','backend' => 'a690a','host' => 'DK-HED42-MK5-Testlab-VOB1-SRV1','service' => 'mssql_database-file-dbcc-shrinks','type' => '2'};
[2021/01/22 12:09:08][dkrdswppthrukp01.vestasext.net][ERROR] user:    nnit-sfpo
[2021/01/22 12:09:08][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.10.39.171
[2021/01/22 12:09:08][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:09:08][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:09:08][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:11:23][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:11:23][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?hidesearch=2&s0_op=~&s0_type=search&add_default_service_filter=1&s0_value=se:ppc_time&reload_nav=1&scrollTo=2478&_=1611313880544
[2021/01/22 12:11:23][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611313880544','add_default_service_filter' => '1','hidesearch' => '2','reload_nav' => '1','s0_op' => '~','s0_type' => 'search','s0_value' => 'se:ppc_time','scrollTo' => '2478'};
[2021/01/22 12:11:23][dkrdswppthrukp01.vestasext.net][ERROR] user:    nnit-drbi
[2021/01/22 12:11:23][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.224.69
[2021/01/22 12:11:23][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:11:23][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:11:23][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?backends=51656&backends=4120b&backends=53dec&backends=d4c78&backends=a103e&backends=bc29d&backends=51d28&backends=bde22&backends=5f10e&backends=a690a&backends=4120b&backends=53dec&backends=d4c78&backends=a103e&backends=bc29d&backends=51d28&backends=bde22&backends=5f10e&backends=a690a&dfl_s0_hostprops=262186&dfl_s0_hoststatustypes=2&dfl_s0_op==&dfl_s0_op=!~&dfl_s0_op=!~&dfl_s0_op=!~&dfl_s0_op=!=&dfl_s0_op=!~&dfl_s0_op=!~&dfl_s0_op=~&dfl_s0_op=!~&dfl_s0_op=>=&dfl_s0_op=!=&dfl_s0_serviceprops=262186&dfl_s0_servicestatustypes=16&dfl_s0_type=custom variable&dfl_s0_type=service&dfl_s0_type=host&dfl_s0_type=service&dfl_s0_type=service&dfl_s0_type=plugin output&dfl_s0_type=service&dfl_s0_type=host&dfl_s0_type=service&dfl_s0_type=duration&dfl_s0_type=service&dfl_s0_val_pre=OPRSTATUS&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_val_pre=&dfl_s0_value=InService&dfl_s0_value=wsus_updates|input_bad_status&dfl_s0_value=-wtg|-wan|-SW|PPC-MAIN&dfl_s0_value=patching_psp_upgrade|interface|envir_|eventlog_|Connectivity&dfl_s0_value=ambient_humidity|ambient_temperature&dfl_s0_value=Connection refused|Plugin timed out|snmpwalk returns no product name|problem connecting to|could not contact snmp agent|unlock_puppet&dfl_s0_value=service_&dfl_s0_value=vob|voc&dfl_s0_value=sql|ntp_|_mem_|task_SCADA_OPC_Agent_Task|task_unlock_puppet&dfl_s0_value=20m&dfl_s0_value=battery_remaining_time&dfl_s1_hostprops=262186&dfl_s1_hoststatustypes=2&dfl_s1_op==&dfl_s1_op=!=&dfl_s1_op=!~&dfl_s1_op=!~&dfl_s1_op=!=&dfl_s1_op=!~&dfl_s1_op=!=&dfl_s1_op=~&dfl_s1_serviceprops=262186&dfl_s1_servicestatustypes=16&dfl_s1_type=custom variable&dfl_s1_type=service&dfl_s1_type=host&dfl_s1_type=service&dfl_s1_type=service&dfl_s1_type=plugin output&dfl_s1_type=service&dfl_s1_type=service&dfl_s1_val_pre=OPRSTATUS&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_val_pre=&dfl_s1_value=InService&dfl_s1_value=wsus_updates&dfl_s1_value=-wtg|-wan|-SW|PPC-MAIN&dfl_s1_value=patching_psp_upgrade|interface|envir_|eventlog_|Connectivity&dfl_s1_value=ambient_humidity|ambient_temperature&dfl_s1_value=Connection refused|Plugin timed out|snmpwalk returns no product name|problem connecting to|could not contact snmp agent&dfl_s1_value=service_3rd_Puppet_Agent&dfl_s1_value=disk_c|disk_d|disk_e|disk_f&dfl_s2_hostprops=262186&dfl_s2_hoststatustypes=2&dfl_s2_op==&dfl_s2_op=!=&dfl_s2_op=!~&dfl_s2_op=!~&dfl_s2_op=!=&dfl_s2_op=!~&dfl_s2_op=!=&dfl_s2_op=~&dfl_s2_op=>=&dfl_s2_serviceprops=262186&dfl_s2_servicestatustypes=16&dfl_s2_type=custom variable&dfl_s2_type=service&dfl_s2_type=host&dfl_s2_type=service&dfl_s2_type=service&dfl_s2_type=plugin output&dfl_s2_type=service&dfl_s2_type=service&dfl_s2_type=duration&dfl_s2_val_pre=OPRSTATUS&dfl_s2_val_pre=&dfl_s2_val_pre=&dfl_s2_val_pre=&dfl_s2_val_pre=&dfl_s2_val_pre=&dfl_s2_val_pre=&dfl_s2_val_pre=&dfl_s2_val_pre=&dfl_s2_value=InService&dfl_s2_value=wsus_updates&dfl_s2_value=-wtg|-wan|-SW|PPC-MAIN&dfl_s2_value=patching_psp_upgrade|interface|envir_|eventlog_|Connectivity&dfl_s2_value=ambient_humidity|ambient_temperature&dfl_s2_value=Connection refused|Plugin timed out|snmpwalk returns no product name|problem connecting to|could not contact snmp agent&dfl_s2_value=service_3rd_Puppet_Agent&dfl_s2_value=sql&dfl_s2_value=1d&dfl_s3_hostprops=262186&dfl_s3_hoststatustypes=2&dfl_s3_op==&dfl_s3_op==&dfl_s3_op=~&dfl_s3_op=!~&dfl_s3_serviceprops=262186&dfl_s3_servicestatustypes=16&dfl_s3_type=custom variable&dfl_s3_type=service&dfl_s3_type=host&dfl_s3_type=host&dfl_s3_val_pre=OPRSTATUS&dfl_s3_val_pre=&dfl_s3_val_pre=&dfl_s3_val_pre=&dfl_s3_value=InService&dfl_s3_value=ntp_timesync&dfl_s3_value=vob|voc|ppc&dfl_s3_value=SW&dfl_s4_hostprops=262186&dfl_s4_hoststatustypes=2&dfl_s4_op==&dfl_s4_op=~&dfl_s4_op==&dfl_s4_serviceprops=262186&dfl_s4_servicestatustypes=16&dfl_s4_type=custom variable&dfl_s4_type=host&dfl_s4_type=service&dfl_s4_val_pre=OPRSTATUS&dfl_s4_val_pre=&dfl_s4_val_pre=&dfl_s4_value=InService&dfl_s4_value=PPC-MAIN&dfl_s4_value=ppc_time&dfl_s5_hostprops=262186&dfl_s5_hoststatustypes=2&dfl_s5_op==&dfl_s5_op=~&dfl_s5_op==&dfl_s5_serviceprops=262186&dfl_s5_servicestatustypes=20&dfl_s5_type=custom variable&dfl_s5_type=host&dfl_s5_type=service&dfl_s5_val_pre=OPRSTATUS&dfl_s5_val_pre=&dfl_s5_val_pre=&dfl_s5_value=InService&dfl_s5_value=PPC-Log&dfl_s5_value=ntp_timesync&dfl_s6_hostprops=262186&dfl_s6_hoststatustypes=2&dfl_s6_op==&dfl_s6_op=~&dfl_s6_op==&dfl_s6_serviceprops=262186&dfl_s6_servicestatustypes=16&dfl_s6_type=custom variable&dfl_s6_type=host&dfl_s6_type=service&dfl_s6_val_pre=OPRSTATUS&dfl_s6_val_pre=&dfl_s6_val_pre=&dfl_s6_value=InService&dfl_s6_value=PPC-Log&dfl_s6_value=connectivity_RDP&scrollTo=800&_=1611313885541
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611313885541','backends' => ['51656','4120b','53dec','d4c78','a103e','bc29d','51d28','bde22','5f10e','a690a','4120b','53dec','d4c78','a103e','bc29d','51d28','bde22','5f10e','a690a'],'dfl_s0_hostprops' => '262186','dfl_s0_hoststatustypes'...
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] user:    nnit-drbi
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.224.69
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] internal lmd error - did not get a valid response for at least any site at /usr/share/thruk/lib/Thruk/Backend/Manager.pm line 1880.
[2021/01/22 12:11:32][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:11:48][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:11:48][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?format=search
[2021/01/22 12:11:48][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'format' => 'search','limit' => '2000','query' => ''};
[2021/01/22 12:11:48][dkrdswppthrukp01.vestasext.net][ERROR] user:    jujrm
[2021/01/22 12:11:48][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.10.48.61
[2021/01/22 12:11:48][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:11:48][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:11:48][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?host=RU-Gukovo-26-VOB1-SRV1&style=detail&dfl_columns=host_name,description,state,last_check,duration,current_attempt,plugin_output,cust_OPRSTATUS&_=1611313908164
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611313908164','dfl_columns' => 'host_name,description,state,last_check,duration,current_attempt,plugin_output,cust_OPRSTATUS','host' => 'RU-Gukovo-26-VOB1-SRV1','style' => 'detail'};
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] user:    snoje
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.10.52.252
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] MED-3: peer is down: connecting... (172.26.66.57:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] NCE-1: peer is down: connecting... (172.26.66.52:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] NCE-3: peer is down: connecting... (172.26.66.53:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] MED-1: peer is down: connecting... (172.26.66.54:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] ASP-1: peer is down: connecting... (172.26.66.55:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] CHI-1: peer is down: connecting... (172.26.66.63:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] AME-1: peer is down: connecting... (172.26.66.56:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] Master: peer is down: connecting... (172.26.66.30:6558)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] NCE-2: peer is down: connecting... (172.26.66.58:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] MED-2: peer is down: connecting... (172.26.66.60:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:11:49][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:13:02][dkrdswppthrukp01.vestasext.net][INFO] [laeda][Master] cmd: COMMAND [1611313982] PROCESS_SERVICE_CHECK_RESULT;RU-Kazachiya-PPC-MAIN;Connectivity;0;newly commissioned|
[2021/01/22 12:13:02][dkrdswppthrukp01.vestasext.net][INFO] [laeda][Master] cmd: COMMAND [1611313982] PROCESS_SERVICE_CHECK_RESULT;RU-Tselinskaya-PPC-LOG-SRV;service_3rd_MongoDB;0;newly commissioned|
[2021/01/22 12:13:02][dkrdswppthrukp01.vestasext.net][INFO] [laeda][Master] cmd: COMMAND [1611313982] PROCESS_SERVICE_CHECK_RESULT;RU-Tselinskaya-PPC-MAIN;Connectivity;0;newly commissioned|
[2021/01/22 12:13:23][dkrdswppthrukp01.vestasext.net][INFO] [laeda][ASP-1] cmd: COMMAND [1611314003] PROCESS_SERVICE_CHECK_RESULT;TH-Romklao-PPC-MAIN;ppc_time;0;PPC Main Down|
[2021/01/22 12:13:32][dkrdswppthrukp01.vestasext.net][INFO] [laeda][ASP-1] cmd: COMMAND [1611314012] SCHEDULE_FORCED_SVC_CHECK;TH-Romklao-PPC-MAIN;ppc_time;1611314012
[2021/01/22 12:13:50][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:13:50][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 15.2s to load.
[2021/01/22 12:13:50][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/panorama.cgi?task=status
[2021/01/22 12:13:50][dkrdswppthrukp01.vestasext.net][WARN] params:  {'current_tab' => 'tabpan-tab_269','dfl_s0_hostprops' => 0,'dfl_s0_hoststatustypes' => 15,'dfl_s0_op' => ['=','=','='],'dfl_s0_serviceprops' => 0,'dfl_s0_servicestatustypes' => 31,'dfl_s0_type' => ['custom variable','service','custom variable'],'d...
[2021/01/22 12:13:50][dkrdswppthrukp01.vestasext.net][WARN] user:    chj
[2021/01/22 12:13:50][dkrdswppthrukp01.vestasext.net][WARN] address: 10.10.48.68
[2021/01/22 12:13:50][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:14:03][dkrdswppthrukp01.vestasext.net][INFO] [laeda][ASP-1] cmd: COMMAND [1611314043] SCHEDULE_FORCED_SVC_CHECK;TH-Romklao-PPC-MAIN;ppc_time;1611314043
[2021/01/22 12:14:04][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:14:04][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?hidesearch=2&s0_op=~&s0_type=search&add_default_service_filter=1&s0_value=se:ppc_time&reload_nav=1&_=1611314041543
[2021/01/22 12:14:04][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611314041543','add_default_service_filter' => '1','hidesearch' => '2','reload_nav' => '1','s0_op' => '~','s0_type' => 'search','s0_value' => 'se:ppc_time'};
[2021/01/22 12:14:04][dkrdswppthrukp01.vestasext.net][ERROR] user:    nnit-drbi
[2021/01/22 12:14:04][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.224.69
[2021/01/22 12:14:04][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:14:04][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:14:04][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:14:10][dkrdswppthrukp01.vestasext.net][INFO] [laeda][ASP-1] cmd: COMMAND [1611314050] SCHEDULE_FORCED_SVC_CHECK;TH-Romklao-PPC-MAIN;ppc_time;1611314050
[2021/01/22 12:14:23][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:14:23][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 16.9s to load.
[2021/01/22 12:14:23][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?hostgroup=all&style=summary&_=1611314044287
[2021/01/22 12:14:23][dkrdswppthrukp01.vestasext.net][WARN] params:  {'_' => '1611314044287','hostgroup' => 'all','style' => 'summary'};
[2021/01/22 12:14:23][dkrdswppthrukp01.vestasext.net][WARN] user:    jfgda
[2021/01/22 12:14:23][dkrdswppthrukp01.vestasext.net][WARN] address: 10.10.61.22
[2021/01/22 12:14:23][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:15:22][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:15:22][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 17.0s to load.
[2021/01/22 12:15:22][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/panorama.cgi?task=status
[2021/01/22 12:15:22][dkrdswppthrukp01.vestasext.net][WARN] params:  {'current_tab' => 'tabpan-tab_269','dfl_s0_hostprops' => 0,'dfl_s0_hoststatustypes' => 15,'dfl_s0_op' => ['=','=','='],'dfl_s0_serviceprops' => 0,'dfl_s0_servicestatustypes' => 31,'dfl_s0_type' => ['custom variable','service','custom variable'],'d...
[2021/01/22 12:15:22][dkrdswppthrukp01.vestasext.net][WARN] user:    chj
[2021/01/22 12:15:22][dkrdswppthrukp01.vestasext.net][WARN] address: 10.10.48.68
[2021/01/22 12:15:22][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:15:45][dkrdswppthrukp01.vestasext.net][INFO] [dmvlk][Master] cmd: COMMAND [1611314145] SCHEDULE_FORCED_SVC_CHECK;NL-GeefsweerLotC-VOC1-SRV;eventlog_Puppet_Agent;1611314145
[2021/01/22 12:16:47][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:16:47][dkrdswppthrukp01.vestasext.net][ERROR] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/status.cgi?hidesearch=2&s0_op=~&s0_type=search&add_default_service_filter=1&s0_value=se:ppc_time&reload_nav=1&_=1611314204548
[2021/01/22 12:16:47][dkrdswppthrukp01.vestasext.net][ERROR] params:  {'_' => '1611314204548','add_default_service_filter' => '1','hidesearch' => '2','reload_nav' => '1','s0_op' => '~','s0_type' => 'search','s0_value' => 'se:ppc_time'};
[2021/01/22 12:16:47][dkrdswppthrukp01.vestasext.net][ERROR] user:    nnit-drbi
[2021/01/22 12:16:47][dkrdswppthrukp01.vestasext.net][ERROR] address: 10.219.224.69
[2021/01/22 12:16:47][dkrdswppthrukp01.vestasext.net][ERROR] No Backend available
[2021/01/22 12:16:47][dkrdswppthrukp01.vestasext.net][ERROR] NCE-4: peer is down: connecting... (172.26.66.59:6557)
[2021/01/22 12:16:47][dkrdswppthrukp01.vestasext.net][ERROR] ***************************
[2021/01/22 12:16:51][dkrdswppthrukp01.vestasext.net][WARN] ***************************
[2021/01/22 12:16:51][dkrdswppthrukp01.vestasext.net][WARN] slow_page_log_threshold (15s) hit, page took 16.7s to load.
[2021/01/22 12:16:51][dkrdswppthrukp01.vestasext.net][WARN] page:    http://wpp-monitoring.vestasext.net/thruk/cgi-bin/panorama.cgi?task=status
[2021/01/22 12:16:51][dkrdswppthrukp01.vestasext.net][WARN] params:  {'current_tab' => 'tabpan-tab_269','dfl_s0_hostprops' => 0,'dfl_s0_hoststatustypes' => 15,'dfl_s0_op' => ['=','=','='],'dfl_s0_serviceprops' => 0,'dfl_s0_servicestatustypes' => 31,'dfl_s0_type' => ['custom variable','service','custom variable'],'d...
[2021/01/22 12:16:51][dkrdswppthrukp01.vestasext.net][WARN] user:    chj
[2021/01/22 12:16:51][dkrdswppthrukp01.vestasext.net][WARN] address: 10.10.48.68
[2021/01/22 12:16:51][dkrdswppthrukp01.vestasext.net][WARN] Profile:
[2021/01/22 12:17:16][dkrdswppthrukp01.vestasext.net][WARN] ***************************
sni commented 3 years ago

Are you doing those tests with the last thruk release? There might have been some changes related to this. Waiting longer for LMD to respond should be pointless or least it simply covers the real issue. In order to determine LMD health status, Thruk sends a small request to the sites table. This table does not block nor does it involve any backend locks so it really should answer immediatly within milliseconds no matter what. In case that does not work, Thruk will send a USR1 signal to make LMD write a thread dump into its log to see if there might be race conditions or anything. In your case i only see the SIGINT but not the SIGUSR1 which is strange.

It looks like this is not a blazing fast restart, but just another LMD starting while the first one is still running. So there are simply two processes writing into the same logfile. On the other hand it is not possible to listen twice on the same socket, so it might be a kill -9 followed immediatly with the start of the next process.

Anyway, i think the real question is why does the health check fails when it seems like everything is fine.

steffen-poulsen commented 3 years ago

Thanks for the input, it makes sense.

And, yeah, this is Thruk 2.32-3, so not the newest version. We will upgrade asap and test again.

There appears to be a correlation between the load coming in from Thruk and the frequency of the these restarts so far - at nighttime and now that it is weekend, everything appears to be stable.

Maybe I can generate some load on our test env and see if I can provoke this error there and try to understand it a bit better.