search

sni / lmd

Livestatus Multitool Daemon - Create livestatus federation from multiple sources
https://labs.consol.de/omd/packages/lmd/
GNU General Public License v3.0
42 stars 31 forks source link

[main.go:680] got sigint, quitting - lmd reloads backends for no apparent reason, sometimes #108

Closed steffen-poulsen closed 3 years ago

steffen-poulsen commented 3 years ago

After we applied the https://github.com/sni/lmd/commit/0980d577687a12de1e53b3252cafa2a848de128f fix, the frequency of this error has increased in our setup.

When it appears, it has the effect that lmd appears to flush all the backends, and the backends need to start over from the beginning. This is seen as red backends in Thruk.

[2021-01-19 07:33:03.111][Info][main.go:678] got sigint, quitting
[2021-01-19 09:36:55.921][Info][main.go:678] got sigint, quitting
[2021-01-19 10:20:56.892][Info][main.go:678] got sigint, quitting
[2021-01-19 14:43:40.910][Info][main.go:678] got sigint, quitting
[2021-01-20 03:41:23.789][Info][main.go:678] got sigint, quitting
[2021-01-20 08:50:03.037][Info][main.go:678] got sigint, quitting
[2021-01-20 11:03:09.320][Info][main.go:678] got sigint, quitting
[2021-01-20 12:28:15.770][Info][main.go:678] got sigint, quitting
[2021-01-20 12:55:48.134][Info][main.go:678] got sigint, quitting
[2021-01-20 15:06:04.177][Info][main.go:678] got sigint, quitting
[2021-01-21 00:35:03.777][Info][main.go:678] got sigint, quitting
[2021-01-21 02:01:09.637][Info][main.go:678] got sigint, quitting
[2021-01-21 10:04:05.032][Info][main.go:678] got sigint, quitting
-- lmd upgrade --
[2021-01-21 11:33:11.370][Info][main.go:680] got sigint, quitting
[2021-01-21 12:02:43.109][Info][main.go:680] got sigint, quitting
[2021-01-21 12:13:40.696][Info][main.go:680] got sigint, quitting
[2021-01-21 12:30:43.110][Info][main.go:680] got sigint, quitting
[2021-01-21 12:44:31.617][Info][main.go:680] got sigint, quitting
[2021-01-21 14:06:51.444][Info][main.go:680] got sigint, quitting
[2021-01-21 14:34:52.465][Info][main.go:680] got sigint, quitting
[2021-01-21 14:46:27.096][Info][main.go:680] got sigint, quitting
[2021-01-21 14:57:50.978][Info][main.go:680] got sigint, quitting

The reloads that fall around :01-04 and :31-34 can be correlated with restarts of the backends.

The reloads that falls outside these intervals appears to just happen spontaneously. For instance the latest reloads at 14:46:27.096 and 14:57:50.978.

The lmd.log reports a varying number of socket error messages when this happens, and then finally starts over.

# cat  /var/log/lmd.log | grep -B 1 -A 30 "got sigint"
[2021-01-21 02:01:07.637][Info][listener.go:165] incoming sites request from @ to /var/cache/thruk/lmd/live.sock finished in 557.912µs, response size: 2.9KiB
[2021-01-21 02:01:09.637][Info][main.go:678] got sigint, quitting
[2021-01-21 02:01:09.637][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 02:01:09.637][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 02:01:09.638][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 02:01:09.638][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 02:01:13.104][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 02:01:13.104][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 02:01:13.105][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 02:01:13.105][Info][peer.go:311] [NCE-4] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [AME-1] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [NCE-1] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [NCE-3] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [MED-1] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [ASP-1] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [APPL-1] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [NCE-2] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [MED-3] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [NAG99] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [MED-2] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [APPL-ChinaHUB] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [CHI-1] starting connection
[2021-01-21 02:01:13.105][Info][peer.go:311] [Master] starting connection
[2021-01-21 02:01:13.487][Info][peer.go:807] [APPL-1] objects created in: 381.724743ms
[2021-01-21 02:01:13.588][Info][peer.go:807] [NAG99] objects created in: 482.185439ms
[2021-01-21 02:01:14.722][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 1.609623ms, response size: 736 B
[2021-01-21 02:01:14.723][Info][listener.go:165] incoming sites request from @ to /var/cache/thruk/lmd/live.sock finished in 302.309µs, response size: 2.0KiB
[2021-01-21 02:01:14.942][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 1.197946ms, response size: 90 B
[2021-01-21 02:01:14.950][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 2.820504ms, response size: 90 B
[2021-01-21 02:01:14.972][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 91.717µs, response size: 90 B
[2021-01-21 02:01:14.978][Info][listener.go:165] incoming hosts request from @ to /var/cache/thruk/lmd/live.sock finished in 128.58µs, response size: 72 B
[2021-01-21 02:01:14.992][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 4.793161ms, response size: 80 B
--
[2021-01-21 10:04:04.937][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 2.56009845s, response size: 69.8MiB
[2021-01-21 10:04:05.032][Info][main.go:678] got sigint, quitting
[2021-01-21 10:04:05.033][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 10:04:05.033][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 10:04:05.033][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 10:04:05.033][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 10:04:05.626][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 10:04:11.008][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 10:04:11.009][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 10:04:11.009][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 10:04:11.009][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 10:04:11.009][Info][peer.go:311] [AME-1] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [NCE-3] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [MED-1] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [NCE-4] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [MED-3] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [CHI-1] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [NCE-2] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [MED-2] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [NAG99] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [APPL-1] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [APPL-ChinaHUB] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [Master] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [NCE-1] starting connection
[2021-01-21 10:04:11.010][Info][peer.go:311] [ASP-1] starting connection
[2021-01-21 10:04:11.097][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 265.294µs, response size: 80 B
[2021-01-21 10:04:11.422][Info][peer.go:807] [APPL-1] objects created in: 411.30088ms
[2021-01-21 10:04:11.505][Info][peer.go:807] [NAG99] objects created in: 494.419291ms
[2021-01-21 10:04:11.596][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 128.071µs, response size: 80 B
[2021-01-21 10:04:11.981][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 234.973µs, response size: 540 B
[2021-01-21 10:04:11.983][Info][listener.go:165] incoming sites request from @ to /var/cache/thruk/lmd/live.sock finished in 336.137µs, response size: 2.0KiB
[2021-01-21 10:04:12.094][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 1.99135ms, response size: 3 B
--
[2021-01-21 11:33:11.299][Info][listener.go:165] incoming hosts request from @ to /var/cache/thruk/lmd/live.sock finished in 51.212563ms, response size: 72 B
[2021-01-21 11:33:11.370][Info][main.go:680] got sigint, quitting
[2021-01-21 11:33:11.370][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:11.371][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 11:33:11.371][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:11.371][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 11:33:11.780][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:11.780][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 11:33:16.858][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:16.858][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:16.859][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:16.859][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 11:33:21.890][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:21.890][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 11:33:22.716][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:22.716][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:22.716][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:22.717][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 11:33:22.717][Info][peer.go:309] [NCE-2] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [CHI-1] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [NCE-1] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [NCE-3] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [MED-2] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [NAG99] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [APPL-ChinaHUB] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [NCE-4] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [MED-1] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [ASP-1] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [APPL-1] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [AME-1] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [MED-3] starting connection
[2021-01-21 11:33:22.717][Info][peer.go:309] [Master] starting connection
--
[2021-01-21 12:02:40.940][Info][listener.go:165] incoming sites request from @ to /var/cache/thruk/lmd/live.sock finished in 393.819µs, response size: 4.5KiB
[2021-01-21 12:02:43.109][Info][main.go:680] got sigint, quitting
[2021-01-21 12:02:43.109][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 12:02:43.109][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 12:02:43.109][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 12:02:43.109][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 12:02:43.496][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:02:43.496][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:02:48.848][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:02:48.849][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:02:48.849][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:02:48.849][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:02:48.849][Info][peer.go:309] [NCE-2] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [MED-1] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [MED-2] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [Master] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [NCE-3] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [AME-1] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [APPL-1] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [APPL-ChinaHUB] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [CHI-1] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [MED-3] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [NCE-1] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [NCE-4] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [ASP-1] starting connection
[2021-01-21 12:02:48.849][Info][peer.go:309] [NAG99] starting connection
[2021-01-21 12:02:49.055][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 292.995µs, response size: 574 B
[2021-01-21 12:02:49.211][Info][peer.go:833] [APPL-1] objects created in: 361.504581ms
[2021-01-21 12:02:49.357][Info][peer.go:833] [NAG99] objects created in: 507.294699ms
[2021-01-21 12:02:49.430][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 146.867µs, response size: 80 B
[2021-01-21 12:02:49.688][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 156.841µs, response size: 736 B
[2021-01-21 12:02:49.690][Info][listener.go:165] incoming sites request from @ to /var/cache/thruk/lmd/live.sock finished in 388.653µs, response size: 2.0KiB
--
[2021-01-21 12:13:38.913][Warn][datastoreset.go:431] [Master] services delta scan timestamp filter too complex: 302
[2021-01-21 12:13:40.696][Info][main.go:680] got sigint, quitting
[2021-01-21 12:13:40.696][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 12:13:40.696][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 12:13:40.696][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 12:13:40.697][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 12:13:41.075][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:13:41.075][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:13:46.181][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:13:46.181][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:13:46.181][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:13:46.182][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:13:46.182][Info][peer.go:309] [ASP-1] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [AME-1] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [NAG99] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [NCE-2] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [MED-2] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [APPL-ChinaHUB] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [Master] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [NCE-3] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [APPL-1] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [MED-1] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [MED-3] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [CHI-1] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [NCE-1] starting connection
[2021-01-21 12:13:46.182][Info][peer.go:309] [NCE-4] starting connection
[2021-01-21 12:13:46.188][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 2.413651ms, response size: 574 B
[2021-01-21 12:13:46.563][Info][peer.go:833] [APPL-1] objects created in: 380.475508ms
[2021-01-21 12:13:46.649][Info][peer.go:833] [NAG99] objects created in: 466.348782ms
[2021-01-21 12:13:47.099][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 56.770704ms, response size: 12.2KiB
[2021-01-21 12:13:47.120][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 5.507899ms, response size: 422 B
[2021-01-21 12:13:48.109][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 324.501µs, response size: 736 B
--
[2021-01-21 12:30:43.059][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 582.821125ms, response size: 5.4KiB
[2021-01-21 12:30:43.110][Info][main.go:680] got sigint, quitting
[2021-01-21 12:30:43.110][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:43.111][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 12:30:43.111][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:43.111][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 12:30:43.569][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:43.569][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:30:48.813][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:48.813][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:48.813][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:48.813][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:30:51.989][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:51.990][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:51.990][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:51.990][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:30:57.160][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:57.161][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:57.161][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:57.161][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:30:59.058][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:59.058][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:59.058][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:59.058][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:30:59.418][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:59.418][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:59.418][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:59.418][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:30:59.418][Info][peer.go:309] [NCE-2] starting connection
[2021-01-21 12:30:59.419][Info][peer.go:309] [NCE-3] starting connection
[2021-01-21 12:30:59.419][Info][peer.go:309] [NCE-4] starting connection
[2021-01-21 12:30:59.419][Info][peer.go:309] [Master] starting connection
--
[2021-01-21 12:44:30.524][Info][listener.go:165] incoming hostgroups request from @ to /var/cache/thruk/lmd/live.sock finished in 1.173094ms, response size: 777 B
[2021-01-21 12:44:31.617][Info][main.go:680] got sigint, quitting
[2021-01-21 12:44:31.617][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:31.617][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:31.617][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 12:44:31.617][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 12:44:31.901][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:31.901][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:44:37.175][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:37.175][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:37.176][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:37.176][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:44:41.280][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.280][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.280][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.280][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:44:41.348][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.348][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.349][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.349][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:44:41.648][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.649][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.649][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:41.649][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:44:42.046][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:42.047][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:42.047][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:42.047][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 12:44:47.295][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:47.295][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:47.296][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 12:44:47.296][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
--
[2021-01-21 14:06:51.286][Info][listener.go:165] incoming hostgroups request from @ to /var/cache/thruk/lmd/live.sock finished in 79.41176ms, response size: 777 B
[2021-01-21 14:06:51.444][Info][main.go:680] got sigint, quitting
[2021-01-21 14:06:51.444][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 14:06:51.444][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 14:06:51.444][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 14:06:51.444][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 14:06:52.007][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:06:52.008][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:06:52.008][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:06:52.008][Info][peer.go:309] [NCE-2] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [NCE-3] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [Master] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [NCE-1] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [NCE-4] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [CHI-1] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [NAG99] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [APPL-ChinaHUB] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [MED-1] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [MED-2] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [ASP-1] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [AME-1] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [MED-3] starting connection
[2021-01-21 14:06:52.008][Info][peer.go:309] [APPL-1] starting connection
[2021-01-21 14:06:52.411][Info][peer.go:833] [APPL-1] objects created in: 402.392444ms
[2021-01-21 14:06:52.484][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 397.113µs, response size: 654 B
[2021-01-21 14:06:52.486][Info][listener.go:165] incoming sites request from @ to /var/cache/thruk/lmd/live.sock finished in 422.794µs, response size: 2.0KiB
[2021-01-21 14:06:52.490][Info][peer.go:833] [NAG99] objects created in: 481.988963ms
[2021-01-21 14:06:53.209][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 160.116µs, response size: 80 B
[2021-01-21 14:06:53.718][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 73.337µs, response size: 80 B
[2021-01-21 14:06:53.949][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 289.36µs, response size: 90 B
[2021-01-21 14:06:53.954][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 117.435µs, response size: 422 B
[2021-01-21 14:06:54.112][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 87.819µs, response size: 80 B
--
[2021-01-21 14:34:51.223][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 246.162114ms, response size: 93 B
[2021-01-21 14:34:52.465][Info][main.go:680] got sigint, quitting
[2021-01-21 14:34:52.465][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 14:34:52.465][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 14:34:52.465][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 14:34:52.465][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 14:34:53.134][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:34:53.136][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 14:34:58.412][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:34:58.413][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:34:58.413][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:34:58.413][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:34:58.413][Info][peer.go:309] [NCE-1] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [AME-1] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [CHI-1] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [APPL-ChinaHUB] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [Master] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [NCE-2] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [NCE-3] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [MED-1] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [MED-3] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [MED-2] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [ASP-1] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [NCE-4] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [NAG99] starting connection
[2021-01-21 14:34:58.413][Info][peer.go:309] [APPL-1] starting connection
[2021-01-21 14:34:58.750][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 369.203µs, response size: 574 B
[2021-01-21 14:34:58.821][Info][peer.go:833] [APPL-1] objects created in: 408.118364ms
[2021-01-21 14:34:58.915][Info][peer.go:833] [NAG99] objects created in: 501.253344ms
[2021-01-21 14:34:58.975][Info][listener.go:165] incoming status request from @ to /var/cache/thruk/lmd/live.sock finished in 214.266µs, response size: 659 B
[2021-01-21 14:34:58.976][Info][listener.go:165] incoming sites request from @ to /var/cache/thruk/lmd/live.sock finished in 286.74µs, response size: 2.0KiB
[2021-01-21 14:34:59.062][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 58.840885ms, response size: 12.2KiB
--
[2021-01-21 14:46:27.035][Info][listener.go:165] incoming services request from @ to /var/cache/thruk/lmd/live.sock finished in 374.091229ms, response size: 97 B
[2021-01-21 14:46:27.096][Info][main.go:680] got sigint, quitting
[2021-01-21 14:46:27.096][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:27.096][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:27.096][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 14:46:27.096][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 14:46:27.429][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:27.429][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 14:46:32.449][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:32.449][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:32.449][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:32.449][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 14:46:35.818][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:35.818][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:35.819][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:35.819][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 14:46:35.948][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:35.948][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:35.948][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:35.948][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:46:35.949][Info][peer.go:309] [MED-2] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [MED-3] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [CHI-1] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [NCE-2] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [MED-1] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [NCE-4] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [NAG99] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [APPL-ChinaHUB] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [NCE-1] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [APPL-1] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [AME-1] starting connection
[2021-01-21 14:46:35.949][Info][peer.go:309] [Master] starting connection
--
[2021-01-21 14:57:50.877][Info][listener.go:165] incoming contactgroups request from @ to /var/cache/thruk/lmd/live.sock finished in 56.707555ms, response size: 528.1KiB
[2021-01-21 14:57:50.978][Info][main.go:680] got sigint, quitting
[2021-01-21 14:57:50.978][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:50.978][Info][listener.go:311] stopping unix listener on /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:50.978][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 14:57:50.978][Info][listener.go:324] unix listener /var/cache/thruk/lmd/live.sock shutdown complete
[2021-01-21 14:57:51.091][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:51.091][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 14:57:56.189][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:56.189][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:56.190][Fatal][listener.go:297] listen error: listen unix /var/cache/thruk/lmd/live.sock: bind: address already in use
[2021-01-21 14:57:59.456][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:59.456][Warn][listener.go:275] removing stale socket: /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:59.456][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:59.457][Info][listener.go:305] listening for incoming queries on unix /var/cache/thruk/lmd/live.sock
[2021-01-21 14:57:59.457][Info][peer.go:309] [ASP-1] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [NAG99] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [MED-1] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [MED-2] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [AME-1] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [MED-3] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [NCE-2] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [NCE-4] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [Master] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [NCE-1] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [CHI-1] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [APPL-ChinaHUB] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [NCE-3] starting connection
[2021-01-21 14:57:59.457][Info][peer.go:309] [APPL-1] starting connection
[2021-01-21 14:57:59.881][Info][peer.go:833] [APPL-1] objects created in: 423.579211ms
[2021-01-21 14:57:59.978][Info][peer.go:833] [NAG99] objects created in: 520.227145ms
[2021-01-21 14:58:00.551][Info][listener.go:165] incoming comments request from @ to /var/cache/thruk/lmd/live.sock finished in 564.769µs, response size: 3 B
sni commented 3 years ago

Please have a look at your thruk.log when a sigint occurs. I assume Thruk does not get a response in time from LMD and tries to restart it.

steffen-poulsen commented 3 years ago

Ah, thanks - perfect correlation :-)

It does look like lmd is serving requests right up until it gets the sigint though. So I wonder what might be the root cause of this.

[2021/01/21 14:57:59][dkrdswppthrukp01.vestasext.net][ERROR] failed to get lmd startup lock: cannot copy file /var/cache/thruk/lmd/startup: No such file or directory at /usr/share/thruk/lib
/Thruk/Utils/IO.pm line 246

We are a few versions behind on Thruk. We will try to upgrade it, and maybe put the lmd socket on tcp instead of the file system and see if that makes a change.

[2021/01/21 00:35:03][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 02:01:09][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 10:04:05][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 11:33:11][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 12:02:43][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 12:13:40][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 12:30:43][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 12:44:31][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 14:06:51][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 14:34:52][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 14:46:27][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 14:57:50][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 15:17:30][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
[2021/01/21 15:34:40][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding, killing with force: rc - -1 -
steffen-poulsen commented 3 years ago

Btw, we noticed that we had the same Listen declaration in both of our lmd.ini files:

# head -1 /var/cache/thruk/lmd/lmd.ini
Listen = ['/var/cache/thruk/lmd/live.sock']

# head -1 /etc/thruk/thruk_local.d/lmd.ini
Listen = ['/var/cache/thruk/lmd/live.sock']

Thruk starts the LMD process like this:

apache 3388 1 99 15:34 ? 05:41:07 /usr/local/bin/lmd -pidfile /var/cache/thruk/lmd/pid -config /var/cache/thruk/lmd/lmd.ini -config /etc/thruk/thruk_local.d/lmd.ini

And it appears the LMD picks up the "/var/cache/thruk/lmd/live.sock" Listen value from both of the files and uses it twice - and the socket war in the log file starts.

Possibly not the root cause of this issue, but at least the log file now looks a lot nicer without the "removing stale socket" warnings :-)

steffen-poulsen commented 3 years ago

I have tried to add a little more patience and log to LMD.pm. It will now allow 30s before restarting the LMD - it was 2s before the change.

To me it seems like the LMD is still running fine when it gets sigint'ed - but let's see what the log has for us tomorrow.

    my $waited = 0;
    my $rc = -1;
    my $rc_shifted = -1;
    while($waited++ < 30 && $rc != 0) {
        POSIX::waitpid($pid, POSIX::WNOHANG);
        $rc = $?;
        $rc_shifted = $? >> 8;
        $c->log->warn("lmd not responding while trying to contact it (waitpid, WNOHANG). Try #" . $waited . " result: \$rc: $rc, \$rc_shifted: $rc_shifted") if $waited > 1;
        sleep(1);
    }
    if($rc != 0) {
        $c->log->warn("lmd not responding, still no response after $waited tries. \$rc: $rc, \$rc_shifted: $rc_shifted");
        $c->log->warn("lmd not responding, killing with force: rc - ".$rc." - ".($! || ""));
        kill('USR1', $lmd_pid);
        kill(2, $pid);
        kill(2, $lmd_pid);
        sleep(1);
        kill(9, $lmd_pid);
        kill(9, $pid);
    }
steffen-poulsen commented 3 years ago

Looking at the LMD log today, we see no more sigint messages - so that is great :-)

Instead, we see the new Thruk waitpid/WNOHANG warnings, when Thruk is unable to get a timely reply from the LMD process.

On one occation the retry count rose to 18 tries, at which point the communication went ok again - but usually it will deem the LMD process OK again on the 3rd try.

So, raising the number of retries from 2 to 3 in Thruk LMD.pm would take away most of the unneeded LMD restarts / backend reloads in our case.

[root@dkrdswppthrukp01 ~]# grep 'Try' /var/log/thruk/thruk.log
[2021/01/21 20:41:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 20:43:45][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 21:32:44][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 21:36:51][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 21:42:14][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 21:43:01][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:01:24][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:02:10][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:05:58][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:14:18][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:17:20][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/21 22:17:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:37:03][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:38:50][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:41:01][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:46:29][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:49:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/21 22:49:22][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:50:54][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:56:29][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/21 22:56:30][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/21 22:56:30][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/21 23:10:49][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 01:03:36][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 01:08:15][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 01:08:16][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 01:09:42][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 01:13:09][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 01:27:53][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 01:33:27][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 01:36:44][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 01:36:45][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 01:40:41][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 02:55:53][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 03:02:02][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 03:03:48][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 03:04:14][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 03:06:01][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 03:14:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:22][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:23][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #4 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:24][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #5 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:25][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #6 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:26][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #7 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:27][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #8 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:28][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #9 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:29][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #10 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:30][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #11 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:31][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #12 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #13 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:33][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #14 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:34][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #15 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:35][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #16 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:36][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #17 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 03:14:37][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #18 result: $rc: 0, $rc_shifted: 0
[2021/01/22 04:23:40][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 04:31:42][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 04:33:40][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 04:34:59][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 04:46:26][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 05:01:48][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 05:03:52][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 05:04:22][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 05:06:59][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 05:33:42][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 05:36:11][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:00:18][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 06:00:19][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:02:59][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:05:13][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:17:47][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:18:52][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:19:16][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:26:01][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:29:59][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:32:45][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 06:55:33][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:03:03][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:03:11][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:04:05][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:05:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:33:08][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:33:09][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:40:31][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:40:46][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:53:54][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:53:57][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:55:42][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 07:55:43][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:56:20][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 07:56:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:56:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:56:45][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:56:49][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 07:58:54][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:01:07][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:02:39][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:02:44][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:05:37][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:06:04][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:06:40][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:08:55][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:16:57][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0                                                                                        [2021/01/22 08:16:58][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:16:58][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:17:01][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:21:29][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:21:33][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:42:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:42:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:44:45][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:45:08][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:45:43][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:45:46][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:49:18][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:49:19][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:52:10][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 08:52:11][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:54:07][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:54:07][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:54:22][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:56:47][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:56:47][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:56:48][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:56:48][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:56:48][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0                                                                                        [2021/01/22 08:56:48][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0                                                                                      [2021/01/22 08:56:49][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 08:57:23][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:02:15][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:02:16][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:02:16][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:03:52][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:21:26][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:26:22][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:30:57][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:31:36][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:32:44][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:39:06][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:41:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:41:35][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 09:41:36][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:50:17][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:50:45][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:51:14][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:51:31][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:51:37][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:54:23][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:54:36][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:54:41][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:55:22][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:56:50][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:57:34][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:57:40][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:57:44][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:57:47][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 09:57:51][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:00:06][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:00:09][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:00:23][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:00:23][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:02:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:02:56][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:02:59][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:03:08][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:06:12][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:06:17][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:27:21][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 10:27:22][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:32:28][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:34:13][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:36:00][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:36:01][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:36:01][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:36:01][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0                                                                                       [2021/01/22 10:36:02][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 10:36:02][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0                                                                                       [2021/01/22 10:36:03][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:36:13][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:36:19][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:38:38][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:41:34][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:42:58][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 10:42:59][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:43:02][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:43:10][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:57:37][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 10:57:37][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 10:57:38][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:57:38][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 10:57:38][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:01:48][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:02:52][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:03:47][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:03:59][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:04:05][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:05:59][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:09:53][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:10:06][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:13:38][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:16:07][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:19:28][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:34:43][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:34:43][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:37:35][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:38:29][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 11:38:30][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:38:30][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:43:00][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:43:04][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:43:10][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:43:13][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:44:43][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:45:16][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:45:36][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:48:13][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 11:48:14][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:48:37][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:48:54][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:49:32][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:52:06][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: -1, $rc_shifted: 72057594037927935
[2021/01/22 11:52:07][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #3 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:52:11][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:52:13][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[2021/01/22 11:54:36][dkrdswppthrukp01.vestasext.net][WARN] lmd not responding while trying to contact it (waitpid, WNOHANG). Try #2 result: $rc: 0, $rc_shifted: 0
[root@dkrdswppthrukp01 ~]# grep 'Try #3' /var/log/thruk/thruk.log | wc
     20     380    3345