search

Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
1.98k stars 570 forks source link

icinga2 segfault #8583

Closed K0nne closed 3 years ago

K0nne commented 3 years ago

Describe the bug

Hello,

last week our icinga-qs configmaster (2.11.7) died. Just before the crash happened, a coleague of mine was testing with downtimes via api.

/var/log/messages

Dec 23 13:31:10 qs-master-01 kernel: [1782953.469945] icinga2[26370]: segfault at fffffffffffffff8 ip 00007f6e0463df7b sp 00007f6dfeb4a430 error 5 in libstdc++.so.6.0.19[7f6e0457f000+e9000]
Dec 23 13:31:10 qs-master-01 kernel: icinga2[26370]: segfault at fffffffffffffff8 ip 00007f6e0463df7b sp 00007f6dfeb4a430 error 5 in libstdc++.so.6.0.19[7f6e0457f000+e9000]
Dec 23 13:31:13 qs-master-01 systemd: icinga2.service: main process exited, code=exited, status=139/n/a
Dec 23 13:31:13 qs-master-01 systemd: Unit icinga2.service entered failed state.
Dec 23 13:31:13 qs-master-01 systemd: icinga2.service failed.

icinga2.log

[2020-12-23 13:30:39 +0100] information/ApiListener: Replayed 1 messages.
[2020-12-23 13:30:39 +0100] information/ApiListener: Finished sending replay log for endpoint 'qs-master-02' in zone 'master'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Finished syncing endpoint 'qs-master-02' in zone 'master'.
[2020-12-23 13:30:39 +0100] information/IdoMysqlConnection: Pending queries: 122 (Input: 8/s; Output: 3/s)
[2020-12-23 13:30:39 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2244, rate:  0/s (0/min 4327/5min 7626/15min); empty in 2 minutes and 48 seconds
[2020-12-23 13:30:39 +0100] information/ApiListener: Reconnecting to endpoint 'qs-master-02' via host 'x.x.x.x' and port '5665'
[2020-12-23 13:30:39 +0100] information/ApiListener: New client connection for identity 'qs-master-02' to [x.x.x.x]:5665
[2020-12-23 13:30:39 +0100] information/ApiListener: Sending config updates for endpoint 'qs-master-02' in zone 'master'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Finished reconnecting to endpoint 'qs-master-02' via host 'x.x.x.x' and port '5665'
[2020-12-23 13:30:39 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Syncing configuration files for zone 'zone-01' to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] warning/JsonRpcConnection: API client disconnected for identity 'qs-master-02'
[2020-12-23 13:30:39 +0100] warning/ApiListener: Removing API client for endpoint 'qs-master-02'. 0 API clients left.
[2020-12-23 13:30:39 +0100] information/ApiListener: Syncing configuration files for zone 'zone-02' to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Syncing configuration files for zone 'zone-03' to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Finished sending config file updates for endpoint 'qs-master-02' in zone 'master'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Syncing runtime objects to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'qs-master-02'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'qs-master-02' in zone 'master'.
[2020-12-23 13:30:39 +0100] information/ApiListener: Sending replay log for endpoint 'qs-master-02' in zone 'master'.
[2020-12-23 13:31:03 +0100] information/ApiListener: New client connection from [x.x.x.x]:44226 (no client certificate)
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/sn5ktest05 (from [x.x.x.x]:44226), user: api-user, agent: Mojolicious (Perl), status: OK).
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:44226)
[2020-12-23 13:31:03 +0100] information/ApiListener: New client connection from [x.x.x.x]:44228 (no client certificate)
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!Interface Usage!service-notifcation-name' of type 'Notification'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!Interface Usage!2296b69e-a21e-4835-a9ec-d7adb28f1eb1' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!ping4!service-notifcation-name' of type 'Notification'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!Interface Usage' of type 'Service'.
[2020-12-23 13:31:03 +0100] information/Checkable: Checkable 'sn5ktest05!CPU' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!CPU!e0ae1a38-b5b3-4c1a-b1d9-7a16ff41f699' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!CPU!service-notifcation-name' of type 'Notification'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!CPU' of type 'Service'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!CPU!e0ae1a38-b5b3-4c1a-b1d9-7a16ff41f699' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!CPU!service-notifcation-name' of type 'Notification'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!ping4!4787bd5c-bad4-441d-9d70-24ba50581c4e' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!host-notifcation-name' of type 'Notification'.
[2020-12-23 13:31:03 +0100] information/Checkable: Checkable 'sn5ktest05' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2020-12-23 13:31:03 +0100] information/Notification: Sending 'DowntimeEnd' notification 'sn5ktest05!downtime-name' for user 'qs-notification-user'
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!78a759b9-bc60-41ad-86fb-168687815c0a' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!downtime-name' of type 'Notification'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05!ping4' of type 'Service'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Deleted object 'sn5ktest05' of type 'Host'.
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: Request: DELETE /v1/objects/hosts/sn5ktest05 (from [x.x.x.x]:44228), user: api-user, agent: Mojolicious (Perl), status: OK).
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:44228)
[2020-12-23 13:31:03 +0100] information/ApiListener: New client connection from [x.x.x.x]:44230 (no client certificate)
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Created and activated object 'sn5ktest05' of type 'Host'.
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: Request: PUT /v1/objects/hosts/sn5ktest05 (from [x.x.x.x]:44230), user: api-user, agent: Mojolicious (Perl), status: OK).
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:44230)
[2020-12-23 13:31:03 +0100] information/ApiListener: New client connection from [x.x.x.x]:44236 (no client certificate)
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/sn5ktest05 (from [x.x.x.x]:44236), user: api-user, agent: Mojolicious (Perl), status: OK).
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:44236)
[2020-12-23 13:31:03 +0100] information/ApiListener: New client connection from [x.x.x.x]:44238 (no client certificate)
[2020-12-23 13:31:03 +0100] information/Downtime: Triggering downtime 'sn5ktest05!67757328-4604-41ea-8405-601040d6ea7f' for checkable 'sn5ktest05'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Created and activated object 'sn5ktest05!67757328-4604-41ea-8405-601040d6ea7f' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/Downtime: Added downtime 'sn5ktest05!67757328-4604-41ea-8405-601040d6ea7f' between '2020-12-23 13:31:03' and '2025-12-23 13:31:03', author: 'icingaweb2-user', fixed
[2020-12-23 13:31:03 +0100] information/Downtime: Triggering downtime 'sn5ktest05!CPU!ce693194-aca4-40ba-8e78-826a7e602413' for checkable 'sn5ktest05!CPU'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Created and activated object 'sn5ktest05!CPU!ce693194-aca4-40ba-8e78-826a7e602413' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/Downtime: Added downtime 'sn5ktest05!CPU!ce693194-aca4-40ba-8e78-826a7e602413' between '2020-12-23 13:31:03' and '2025-12-23 13:31:03', author: 'icingaweb2-user', fixed
[2020-12-23 13:31:03 +0100] information/Downtime: Triggering downtime 'sn5ktest05!Interface Usage!cf5bbc30-0f92-43fe-b8fd-09a3da5a2994' for checkable 'sn5ktest05!Interface Usage'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Created and activated object 'sn5ktest05!Interface Usage!cf5bbc30-0f92-43fe-b8fd-09a3da5a2994' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/Downtime: Added downtime 'sn5ktest05!Interface Usage!cf5bbc30-0f92-43fe-b8fd-09a3da5a2994' between '2020-12-23 13:31:03' and '2025-12-23 13:31:03', author: 'icingaweb2-user', fixed
[2020-12-23 13:31:03 +0100] information/Downtime: Triggering downtime 'sn5ktest05!ping4!019cc082-a9b6-4fe4-a52a-0ea4a91f88ba' for checkable 'sn5ktest05!ping4'.
[2020-12-23 13:31:03 +0100] information/ConfigObjectUtility: Created and activated object 'sn5ktest05!ping4!019cc082-a9b6-4fe4-a52a-0ea4a91f88ba' of type 'Downtime'.
[2020-12-23 13:31:03 +0100] information/Downtime: Added downtime 'sn5ktest05!ping4!019cc082-a9b6-4fe4-a52a-0ea4a91f88ba' between '2020-12-23 13:31:03' and '2025-12-23 13:31:03', author: 'icingaweb2-user', fixed
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime (from [x.x.x.x]:44238), user: api-user, agent: Mojolicious (Perl), status: OK).
[2020-12-23 13:31:03 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:44238)
[2020-12-23 13:31:10 +0100] information/ApiListener: Replayed 1 messages.
[2020-12-23 13:31:10 +0100] information/ApiListener: Finished sending replay log for endpoint 'qs-master-02' in zone 'master'.
[2020-12-23 13:31:10 +0100] information/ApiListener: Finished syncing endpoint 'qs-master-02' in zone 'master'.

To Reproduce

Unknown

Expected behavior

Icinga should not die.

Your Environment

Include as many relevant details about the environment you experienced the problem in

widhalmt commented 3 years ago

ref/NC/699401

julianbrost commented 3 years ago

When I first looked at the logs I missed the line recreating the host somewhere in the middle and tried to reproduce the crash by scheduling downtimes in parallel to deleting the corresponding host and services. This turned out to be successful and now I'm wondering if that's actually the same problem or possibly only a related one.

@K0nne From the log I can tell that you create, fetch, delete hosts and schedule downtimes using the API. Are some of these requests performed in parallel, especially delete host and schedule downtime?

julianbrost commented 3 years ago

Backtrace for my crash (master branch):

Thread 575 "icinga2" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f1e7f6fd700 (LWP 1488)]
__GI___pthread_mutex_lock (mutex=mutex@entry=0x588) at ../nptl/pthread_mutex_lock.c:67
67  ../nptl/pthread_mutex_lock.c: No such file or directory.
(gdb) bt
#0  __GI___pthread_mutex_lock (mutex=mutex@entry=0x588) at ../nptl/pthread_mutex_lock.c:67
#1  0x000055eb589f92a8 in boost::posix::pthread_mutex_lock (m=0x588) at /usr/include/boost/thread/pthread/mutex.hpp:58
#2  boost::mutex::lock (this=0x588) at /usr/include/boost/thread/pthread/mutex.hpp:112
#3  0x000055eb589f934c in boost::unique_lock<boost::mutex>::lock (this=0x7f1e7f6fc590) at /usr/include/boost/thread/lock_types.hpp:334
#4  0x000055eb58bfaaf6 in boost::unique_lock<boost::mutex>::unique_lock (m_=..., this=0x7f1e7f6fc590) at /usr/include/boost/thread/lock_types.hpp:121
#5  icinga::Host::GetServiceByShortName (this=0x0, name=...) at /i2cp/lib/icinga/host.cpp:122
#6  0x000055eb58bfcaae in icinga::Downtime::OnAllConfigLoaded (this=0x7f1de80a3af0) at /i2cp/lib/base/atomic.hpp:52
#7  0x000055eb58a6b7c2 in icinga::ConfigItem::<lambda(const ItemPair&)>::operator()(const icinga::ConfigItem::<lambda(const ItemPair&)>::ItemPair &) (ip={...}, __closure=<optimized out>, __closure=<optimized out>)
    at /i2cp/lib/config/configitem.cpp:525
#8  0x000055eb58997618 in std::function<void ()>::operator()() const (this=this@entry=0x7f1e7f6fca50) at /usr/include/c++/8/bits/std_function.h:682
#9  icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) (this=this@entry=0x7f1e44096ec0, func=...) at /i2cp/lib/base/workqueue.cpp:236
#10 0x000055eb58a63d9b in icinga::WorkQueue::<lambda()>::operator() (this=<optimized out>) at /usr/include/c++/8/bits/std_function.h:87
#11 std::_Function_handler<void(), icinga::WorkQueue::ParallelFor(const VectorType&, const FuncType&) [with VectorType = std::vector<std::pair<boost::intrusive_ptr<icinga::ConfigItem>, bool> >; FuncType = icinga::ConfigItem::CommitNewItems(const Ptr&, icinga::WorkQueue&, std::vector<boost::intrusive_ptr<icinga::ConfigItem> >&)::<lambda(const ItemPair&)>]::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/8/bits/std_function.h:297
#12 0x000055eb58997618 in std::function<void ()>::operator()() const (this=<optimized out>) at /usr/include/c++/8/bits/std_function.h:682
#13 icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) (this=0x7f1e44096ec0, func=...) at /i2cp/lib/base/workqueue.cpp:236
#14 0x000055eb58997e40 in icinga::WorkQueue::WorkerThreadProc (this=<optimized out>) at /i2cp/lib/base/workqueue.cpp:279
#15 0x00007f1ea76ef615 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.67.0
#16 0x00007f1ea76bffa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#17 0x00007f1ea6d384cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
julianbrost commented 3 years ago

https://github.com/Icinga/icinga2/blob/cb25be2d12d1b9a68ec4be85ace2f2b267f875fe/lib/icinga/downtime.cpp#L83

Another crash after quickly fixing this (host may be null):

Thread 30 "icinga2" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f8a8beb9700 (LWP 130)]
icinga::ApiActions::ScheduleDowntime (object=..., params=...) at /i2cp/lib/icinga/apiactions.cpp:391
391 /i2cp/lib/icinga/apiactions.cpp: No such file or directory.
(gdb) bt
#0  icinga::ApiActions::ScheduleDowntime (object=..., params=...) at /i2cp/lib/icinga/apiactions.cpp:391
#1  0x0000562f400d41cc in std::_Function_handler<icinga::Value (boost::intrusive_ptr<icinga::ConfigObject> const&, boost::intrusive_ptr<icinga::Dictionary> const&), boost::intrusive_ptr<icinga::Dictionary> (*)(boost::intrusive_ptr<icinga::ConfigObject> const&, boost::intrusive_ptr<icinga::Dictionary> const&)>::_M_invoke(std::_Any_data const&, boost::intrusive_ptr<icinga::ConfigObject> const&, boost::intrusive_ptr<icinga::Dictionary> const&) (__functor=..., __args#0=..., 
    __args#1=...) at /usr/include/c++/8/bits/std_function.h:88
#2  0x0000562f3ff114e0 in std::function<icinga::Value (boost::intrusive_ptr<icinga::ConfigObject> const&, boost::intrusive_ptr<icinga::Dictionary> const&)>::operator()(boost::intrusive_ptr<icinga::ConfigObject> const&, boost::intrusive_ptr<icinga::Dictionary> const&) const (__args#1=..., __args#0=..., this=<optimized out>) at /usr/include/c++/8/bits/std_function.h:682
#3  icinga::ApiAction::Invoke (params=..., target=..., this=<optimized out>) at /i2cp/lib/remote/apiaction.cpp:14
#4  icinga::ActionsHandler::HandleRequest (this=<optimized out>, stream=..., user=..., request=..., url=..., response=..., params=..., yc=..., server=...) at /i2cp/lib/remote/actionshandler.cpp:87
#5  0x0000562f3ff0e653 in icinga::HttpHandler::ProcessRequest (stream=..., user=..., request=..., response=..., yc=..., server=...) at /i2cp/lib/remote/httphandler.cpp:111
#6  0x0000562f3ff3a29c in ProcessRequest (yc=..., hasStartedStreaming=@0x7f8a5c0849d1: false, server=..., response=..., authenticatedUser=..., request=..., stream=...) at /i2cp/lib/remote/httpserverconnection.cpp:456
#7  icinga::HttpServerConnection::ProcessMessages (this=0x7f8a5c084950, yc=...) at /i2cp/lib/remote/httpserverconnection.cpp:571
#8  0x0000562f3ff3b61e in icinga::HttpServerConnection::<lambda(boost::asio::yield_context)>::operator() (__closure=<optimized out>, yc=...) at /i2cp/lib/remote/httpserverconnection.cpp:66
#9  icinga::IoEngine::<lambda(boost::asio::yield_context)>::operator()(boost::asio::yield_context) const (this=<optimized out>, yc=...) at /i2cp/lib/base/io-engine.hpp:106
#10 0x0000562f3ff3b782 in boost::asio::detail::coro_entry_point<boost::asio::executor_binder<void (*)(), boost::asio::io_context::strand>, icinga::IoEngine::SpawnCoroutine(Handler&, Function) [with Handler = boost::asio::io_context::strand; Function = icinga::HttpServerConnection::Start()::<lambda(boost::asio::yield_context)>]::<lambda(boost::asio::yield_context)> >::operator() (ca=..., this=<optimized out>) at /usr/include/boost/asio/io_context_strand.hpp:113
#11 boost::coroutines::detail::push_coroutine_object<boost::coroutines::pull_coroutine<void>, void, boost::asio::detail::coro_entry_point<boost::asio::executor_binder<void (*)(), boost::asio::io_context::strand>, icinga::IoEngine::SpawnCoroutine(Handler&, Function) [with Handler = boost::asio::io_context::strand; Function = icinga::HttpServerConnection::Start()::<lambda(boost::asio::yield_context)>]::<lambda(boost::asio::yield_context)> >&, boost::coroutines::basic_standard_stack_allocator<boost::coroutines::stack_traits> >::run (this=0x7f8a60293400) at /usr/include/boost/coroutine/detail/push_coroutine_object.hpp:293
#12 boost::coroutines::detail::trampoline_push_void<boost::coroutines::detail::push_coroutine_object<boost::coroutines::pull_coroutine<void>, void, boost::asio::detail::coro_entry_point<boost::asio::executor_binder<void (*)(), boost::asio::io_context::strand>, icinga::IoEngine::SpawnCoroutine(Handler&, Function) [with Handler = boost::asio::io_context::strand; Function = icinga::HttpServerConnection::Start()::<lambda(boost::asio::yield_context)>]::<lambda(boost::asio::yield_context)> >&, boost::coroutines::basic_standard_stack_allocator<boost::coroutines::stack_traits> > >(boost::context::detail::transfer_t) (t=...) at /usr/include/boost/coroutine/detail/trampoline_push.hpp:70
#13 0x00007f8ab39e51ef in make_fcontext () from /usr/lib/x86_64-linux-gnu/libboost_context.so.1.67.0
#14 0x0000562f404814d8 in vtable for boost::coroutines::detail::push_coroutine_object<boost::coroutines::pull_coroutine<void>, void, boost::asio::detail::coro_entry_point<boost::asio::executor_binder<void (*)(), boost::asio::io_context::strand>, icinga::IoEngine::SpawnCoroutine<boost::asio::io_context::strand, icinga::HttpServerConnection::Start()::{lambda(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >)#1}>(boost::asio::io_context::strand&, icinga::HttpServerConnection::Start()::{lambda(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >)#1})::{lambda(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >)#1}>&, boost::coroutines::basic_standard_stack_allocator<boost::coroutines::stack_traits> > ()
#15 0x0000000000000026 in ?? ()
#16 0x0000000000000000 in ?? ()
K0nne commented 3 years ago

@julianbrost I talked with my colleague.

The origin of those CIs is an external CMDB. The CIs are added with the Icinga API. New hosts are simply added. If hosts attributes change, the host in question is deleted and re-created under the same name with updated attributes.

Downtimes for hosts and all of their services are handled independent, based on the "device state" in the external CMDB.

The approach for re-creation:

  1. Query host to verify its existence
  2. Delete host and everything around it (the host, its services and downtimes) with cascade=1
  3. Query the deleted host to verity its gone
  4. re-create host with updated attributes

(After every API call the result is checked for HTTP 200.)

  1. If the new host has a matching external "host state", a downtime for the host and all of its services is applied.

Right now this sequence is only executed manually.

K0nne commented 3 years ago

We reproduced it (CI: testswitch4). Here are the logs of both masters from object creation until the crash occured:

QS_MASTER_01 - /var/log/icinga2/icinga2.log

[2021-01-08 15:16:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:39050 (no client certificate)
[2021-01-08 15:16:11 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39050), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: Not Found).
[2021-01-08 15:16:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39050)
[2021-01-08 15:16:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:39052 (no client certificate)
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:16:11 +0100] information/HttpServerConnection: Request: PUT /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39052), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:16:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39052)
[2021-01-08 15:16:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:39058 (no client certificate)
[2021-01-08 15:16:11 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39058), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:16:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39058)
[2021-01-08 15:16:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:39060 (no client certificate)
[2021-01-08 15:16:11 +0100] information/Downtime: Triggering downtime 'testswitch4!1427520e-2c49-4f47-9a84-3754cf6dd7b5' for checkable 'testswitch4'.
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!1427520e-2c49-4f47-9a84-3754cf6dd7b5' of type 'Downtime'.
[2021-01-08 15:16:11 +0100] information/Downtime: Added downtime 'testswitch4!1427520e-2c49-4f47-9a84-3754cf6dd7b5' between '2021-01-08 15:16:11' and '2026-01-08 15:16:11', author: 'API_USER', fixed
[2021-01-08 15:16:11 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!cb3de749-46d8-48ba-89d1-39923eb27e0e' for checkable 'testswitch4!CPU'.
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!cb3de749-46d8-48ba-89d1-39923eb27e0e' of type 'Downtime'.
[2021-01-08 15:16:11 +0100] information/Downtime: Added downtime 'testswitch4!CPU!cb3de749-46d8-48ba-89d1-39923eb27e0e' between '2021-01-08 15:16:11' and '2026-01-08 15:16:11', author: 'API_USER', fixed
[2021-01-08 15:16:11 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!e4ffd2b2-1679-490f-958f-7f2769b2a22d' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!e4ffd2b2-1679-490f-958f-7f2769b2a22d' of type 'Downtime'.
[2021-01-08 15:16:11 +0100] information/Downtime: Added downtime 'testswitch4!Interface Usage!e4ffd2b2-1679-490f-958f-7f2769b2a22d' between '2021-01-08 15:16:11' and '2026-01-08 15:16:11', author: 'API_USER', fixed
[2021-01-08 15:16:11 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!9173bd81-4fbc-43e9-b624-3635010ad719' for checkable 'testswitch4!ping4'.
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!9173bd81-4fbc-43e9-b624-3635010ad719' of type 'Downtime'.
[2021-01-08 15:16:11 +0100] information/Downtime: Added downtime 'testswitch4!ping4!9173bd81-4fbc-43e9-b624-3635010ad719' between '2021-01-08 15:16:11' and '2026-01-08 15:16:11', author: 'API_USER', fixed
[2021-01-08 15:16:11 +0100] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime (from [x.x.x.x]:39060), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:16:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39060)
[2021-01-08 15:16:33 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.66667/s (160/min 806/5min 2433/15min);
[2021-01-08 15:16:43 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.88333/s (173/min 808/5min 2434/15min);
[2021-01-08 15:16:53 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.81667/s (169/min 815/5min 2436/15min);
[2021-01-08 15:17:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:39150 (no client certificate)
[2021-01-08 15:17:11 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39150), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:17:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39150)
[2021-01-08 15:17:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:39152 (no client certificate)
[2021-01-08 15:17:11 +0100] information/HttpServerConnection: Request: POST /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39152), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:17:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39152)
[2021-01-08 15:17:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:39154 (no client certificate)
[2021-01-08 15:17:11 +0100] information/HttpServerConnection: Request: GET /v1/objects/downtimes (from [x.x.x.x]:39154), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:17:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39154)
[2021-01-08 15:17:13 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.86667/s (172/min 815/5min 2440/15min);
[2021-01-08 15:17:22 +0100] information/ApiListener: New client connection from [x.x.x.x]:39224 (no client certificate)
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39224), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39224)
[2021-01-08 15:17:22 +0100] information/ApiListener: New client connection from [x.x.x.x]:39226 (no client certificate)
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!e4ffd2b2-1679-490f-958f-7f2769b2a22d' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!e4ffd2b2-1679-490f-958f-7f2769b2a22d' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!cb3de749-46d8-48ba-89d1-39923eb27e0e' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME_HOST' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!1427520e-2c49-4f47-9a84-3754cf6dd7b5' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!9173bd81-4fbc-43e9-b624-3635010ad719' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: Request: DELETE /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39226), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39226)
[2021-01-08 15:17:22 +0100] information/ApiListener: New client connection from [x.x.x.x]:39228 (no client certificate)
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: Request: PUT /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39228), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39228)
[2021-01-08 15:17:22 +0100] information/ApiListener: New client connection from [x.x.x.x]:39234 (no client certificate)
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39234), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39234)
[2021-01-08 15:17:22 +0100] information/ApiListener: New client connection from [x.x.x.x]:39236 (no client certificate)
[2021-01-08 15:17:22 +0100] information/Downtime: Triggering downtime 'testswitch4!14300750-5d1f-43ce-89bf-8592ef6afca3' for checkable 'testswitch4'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!14300750-5d1f-43ce-89bf-8592ef6afca3' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Downtime: Added downtime 'testswitch4!14300750-5d1f-43ce-89bf-8592ef6afca3' between '2021-01-08 15:17:22' and '2026-01-08 15:17:22', author: 'API_USER', fixed
[2021-01-08 15:17:22 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!d9ed52fc-d35f-4c09-aa12-f6fff1b80987' for checkable 'testswitch4!CPU'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!d9ed52fc-d35f-4c09-aa12-f6fff1b80987' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Downtime: Added downtime 'testswitch4!CPU!d9ed52fc-d35f-4c09-aa12-f6fff1b80987' between '2021-01-08 15:17:22' and '2026-01-08 15:17:22', author: 'API_USER', fixed
[2021-01-08 15:17:22 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!6156e0f1-621a-4df7-ba86-320cfd8113e5' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!6156e0f1-621a-4df7-ba86-320cfd8113e5' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Downtime: Added downtime 'testswitch4!Interface Usage!6156e0f1-621a-4df7-ba86-320cfd8113e5' between '2021-01-08 15:17:22' and '2026-01-08 15:17:22', author: 'API_USER', fixed
[2021-01-08 15:17:22 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!dc9bad9b-27c8-40c9-bcb9-8f0ae6dac741' for checkable 'testswitch4!ping4'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!dc9bad9b-27c8-40c9-bcb9-8f0ae6dac741' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Downtime: Added downtime 'testswitch4!ping4!dc9bad9b-27c8-40c9-bcb9-8f0ae6dac741' between '2021-01-08 15:17:22' and '2026-01-08 15:17:22', author: 'API_USER', fixed
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime (from [x.x.x.x]:39236), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:17:22 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39236)
[2021-01-08 15:17:23 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate:  3/s (180/min 812/5min 2445/15min);
[2021-01-08 15:17:27 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2021-01-08 15:17:33 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 3.03333/s (182/min 810/5min 2434/15min);
[2021-01-08 15:17:43 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 3.01667/s (181/min 818/5min 2440/15min);
[2021-01-08 15:17:50 +0100] information/ApiListener: New client connection from [127.0.0.1]:60610 (no client certificate)
[2021-01-08 15:17:50 +0100] information/HttpServerConnection: Request: POST /v1/actions/reschedule-check (from [127.0.0.1]:60610), user: icingaweb2, agent: , status: OK).
[2021-01-08 15:17:50 +0100] information/HttpServerConnection: HTTP client disconnected (from [127.0.0.1]:60610)
[2021-01-08 15:18:03 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.95/s (177/min 821/5min 2449/15min);
[2021-01-08 15:18:13 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.88333/s (173/min 815/5min 2442/15min);
[2021-01-08 15:18:17 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2021-01-08 15:18:23 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.63333/s (158/min 807/5min 2436/15min);
[2021-01-08 15:18:26 +0100] information/ApiListener: New client connection from [x.x.x.x]:39324 (no client certificate)
[2021-01-08 15:18:26 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39324), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:26 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39324)
[2021-01-08 15:18:27 +0100] information/ApiListener: New client connection from [x.x.x.x]:39326 (no client certificate)
[2021-01-08 15:18:27 +0100] information/HttpServerConnection: Request: GET /v1/objects/downtimes (from [x.x.x.x]:39326), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:27 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39326)
[2021-01-08 15:18:33 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.55/s (153/min 810/5min 2439/15min);
[2021-01-08 15:18:35 +0100] information/ApiListener: New client connection from [x.x.x.x]:39394 (no client certificate)
[2021-01-08 15:18:35 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39394), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:35 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39394)
[2021-01-08 15:18:35 +0100] information/ApiListener: New client connection from [x.x.x.x]:39396 (no client certificate)
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!d9ed52fc-d35f-4c09-aa12-f6fff1b80987' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!6156e0f1-621a-4df7-ba86-320cfd8113e5' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!d9ed52fc-d35f-4c09-aa12-f6fff1b80987' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/Checkable: Checkable 'testswitch4' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!14300750-5d1f-43ce-89bf-8592ef6afca3' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME_HOST' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!dc9bad9b-27c8-40c9-bcb9-8f0ae6dac741' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:18:35 +0100] information/HttpServerConnection: Request: DELETE /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39396), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:35 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39396)
[2021-01-08 15:18:35 +0100] information/ApiListener: New client connection from [x.x.x.x]:39398 (no client certificate)
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:18:35 +0100] information/HttpServerConnection: Request: PUT /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39398), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:35 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39398)
[2021-01-08 15:18:36 +0100] information/ApiListener: New client connection from [x.x.x.x]:39404 (no client certificate)
[2021-01-08 15:18:36 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39404), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:36 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39404)
[2021-01-08 15:18:36 +0100] information/ApiListener: New client connection from [x.x.x.x]:39406 (no client certificate)
[2021-01-08 15:18:36 +0100] information/Downtime: Triggering downtime 'testswitch4!9716a401-1d6f-4961-958b-c3bcbe426384' for checkable 'testswitch4'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!9716a401-1d6f-4961-958b-c3bcbe426384' of type 'Downtime'.
[2021-01-08 15:18:36 +0100] information/Downtime: Added downtime 'testswitch4!9716a401-1d6f-4961-958b-c3bcbe426384' between '2021-01-08 15:18:35' and '2026-01-08 15:18:35', author: 'API_USER', fixed
[2021-01-08 15:18:36 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!54cc4fc1-a911-4891-a8b5-0e7ea13d1fe0' for checkable 'testswitch4!CPU'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!54cc4fc1-a911-4891-a8b5-0e7ea13d1fe0' of type 'Downtime'.
[2021-01-08 15:18:36 +0100] information/Downtime: Added downtime 'testswitch4!CPU!54cc4fc1-a911-4891-a8b5-0e7ea13d1fe0' between '2021-01-08 15:18:35' and '2026-01-08 15:18:35', author: 'API_USER', fixed
[2021-01-08 15:18:36 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!0997428b-730c-4a0f-8038-208bc84f5bbe' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!0997428b-730c-4a0f-8038-208bc84f5bbe' of type 'Downtime'.
[2021-01-08 15:18:36 +0100] information/Downtime: Added downtime 'testswitch4!Interface Usage!0997428b-730c-4a0f-8038-208bc84f5bbe' between '2021-01-08 15:18:35' and '2026-01-08 15:18:35', author: 'API_USER', fixed
[2021-01-08 15:18:36 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!ac3ab9c8-ad76-4347-aaf2-dc87930f9082' for checkable 'testswitch4!ping4'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!ac3ab9c8-ad76-4347-aaf2-dc87930f9082' of type 'Downtime'.
[2021-01-08 15:18:36 +0100] information/Downtime: Added downtime 'testswitch4!ping4!ac3ab9c8-ad76-4347-aaf2-dc87930f9082' between '2021-01-08 15:18:35' and '2026-01-08 15:18:35', author: 'API_USER', fixed
[2021-01-08 15:18:36 +0100] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime (from [x.x.x.x]:39406), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:36 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39406)
[2021-01-08 15:18:47 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2021-01-08 15:18:53 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.5/s (150/min 815/5min 2438/15min);
[2021-01-08 15:18:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39474 (no client certificate)
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39474), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39474)
[2021-01-08 15:18:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39476 (no client certificate)
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!0997428b-730c-4a0f-8038-208bc84f5bbe' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!54cc4fc1-a911-4891-a8b5-0e7ea13d1fe0' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Checkable: Checkable 'testswitch4' has 2 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!9716a401-1d6f-4961-958b-c3bcbe426384' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME_HOST' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!0997428b-730c-4a0f-8038-208bc84f5bbe' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!ac3ab9c8-ad76-4347-aaf2-dc87930f9082' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!ac3ab9c8-ad76-4347-aaf2-dc87930f9082' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: Request: DELETE /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39476), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39476)
[2021-01-08 15:18:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39478 (no client certificate)
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: Request: PUT /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39478), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39478)
[2021-01-08 15:18:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39484 (no client certificate)
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39484), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39484)
[2021-01-08 15:18:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39486 (no client certificate)
[2021-01-08 15:18:54 +0100] information/Downtime: Triggering downtime 'testswitch4!3a24d944-e233-43fd-868f-18bac207ad68' for checkable 'testswitch4'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!3a24d944-e233-43fd-868f-18bac207ad68' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Downtime: Added downtime 'testswitch4!3a24d944-e233-43fd-868f-18bac207ad68' between '2021-01-08 15:18:54' and '2026-01-08 15:18:54', author: 'API_USER', fixed
[2021-01-08 15:18:54 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!d20edbd0-172b-4a45-8630-6c8bdde73819' for checkable 'testswitch4!CPU'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!d20edbd0-172b-4a45-8630-6c8bdde73819' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Downtime: Added downtime 'testswitch4!CPU!d20edbd0-172b-4a45-8630-6c8bdde73819' between '2021-01-08 15:18:54' and '2026-01-08 15:18:54', author: 'API_USER', fixed
[2021-01-08 15:18:54 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!fe40edf5-b121-4db7-97af-8be820ca7268' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!fe40edf5-b121-4db7-97af-8be820ca7268' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Downtime: Added downtime 'testswitch4!Interface Usage!fe40edf5-b121-4db7-97af-8be820ca7268' between '2021-01-08 15:18:54' and '2026-01-08 15:18:54', author: 'API_USER', fixed
[2021-01-08 15:18:54 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!32e363f2-6dc7-4017-ac88-6474d06e6769' for checkable 'testswitch4!ping4'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!32e363f2-6dc7-4017-ac88-6474d06e6769' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Downtime: Added downtime 'testswitch4!ping4!32e363f2-6dc7-4017-ac88-6474d06e6769' between '2021-01-08 15:18:54' and '2026-01-08 15:18:54', author: 'API_USER', fixed
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime (from [x.x.x.x]:39486), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:18:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39486)
[2021-01-08 15:18:56 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-01-08 15:19:03 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.48333/s (149/min 821/5min 2450/15min);
[2021-01-08 15:19:13 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.41667/s (145/min 809/5min 2444/15min);
[2021-01-08 15:19:23 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.75/s (165/min 813/5min 2446/15min);
[2021-01-08 15:19:43 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.46667/s (148/min 818/5min 2438/15min);
[2021-01-08 15:19:46 +0100] information/ApiListener: New client connection from [x.x.x.x]:39572 (no client certificate)
[2021-01-08 15:19:46 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39572), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:19:46 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39572)
[2021-01-08 15:19:46 +0100] information/ApiListener: New client connection from [x.x.x.x]:39574 (no client certificate)
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!d20edbd0-172b-4a45-8630-6c8bdde73819' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!fe40edf5-b121-4db7-97af-8be820ca7268' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!fe40edf5-b121-4db7-97af-8be820ca7268' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!32e363f2-6dc7-4017-ac88-6474d06e6769' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!d20edbd0-172b-4a45-8630-6c8bdde73819' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME_HOST' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!3a24d944-e233-43fd-868f-18bac207ad68' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:19:46 +0100] information/HttpServerConnection: Request: DELETE /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39574), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:19:46 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39574)
[2021-01-08 15:19:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39654 (no client certificate)
[2021-01-08 15:19:54 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39654), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: Not Found).
[2021-01-08 15:19:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39654)
[2021-01-08 15:19:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39656 (no client certificate)
[2021-01-08 15:19:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:19:54 +0100] information/HttpServerConnection: Request: PUT /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39656), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:19:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39656)
[2021-01-08 15:19:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39662 (no client certificate)
[2021-01-08 15:19:54 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39662), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:19:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39662)
[2021-01-08 15:19:54 +0100] information/ApiListener: New client connection from [x.x.x.x]:39664 (no client certificate)
[2021-01-08 15:19:54 +0100] information/Downtime: Triggering downtime 'testswitch4!40ffbde6-a9e6-4c56-96ca-5f57c481efd4' for checkable 'testswitch4'.
[2021-01-08 15:19:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!40ffbde6-a9e6-4c56-96ca-5f57c481efd4' of type 'Downtime'.
[2021-01-08 15:19:54 +0100] information/Downtime: Added downtime 'testswitch4!40ffbde6-a9e6-4c56-96ca-5f57c481efd4' between '2021-01-08 15:19:54' and '2026-01-08 15:19:54', author: 'API_USER', fixed
[2021-01-08 15:19:54 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!570a0584-d3aa-4f1d-be48-ccd58b484396' for checkable 'testswitch4!CPU'.
[2021-01-08 15:19:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!570a0584-d3aa-4f1d-be48-ccd58b484396' of type 'Downtime'.
[2021-01-08 15:19:54 +0100] information/Downtime: Added downtime 'testswitch4!CPU!570a0584-d3aa-4f1d-be48-ccd58b484396' between '2021-01-08 15:19:54' and '2026-01-08 15:19:54', author: 'API_USER', fixed
[2021-01-08 15:19:54 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!e3647339-27a8-40d9-bb91-b0333d31e9f9' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:19:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!e3647339-27a8-40d9-bb91-b0333d31e9f9' of type 'Downtime'.
[2021-01-08 15:19:54 +0100] information/Downtime: Added downtime 'testswitch4!Interface Usage!e3647339-27a8-40d9-bb91-b0333d31e9f9' between '2021-01-08 15:19:54' and '2026-01-08 15:19:54', author: 'API_USER', fixed
[2021-01-08 15:19:54 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!36b1ad18-c5b6-4654-92d8-e2e8ceb7aac4' for checkable 'testswitch4!ping4'.
[2021-01-08 15:19:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!36b1ad18-c5b6-4654-92d8-e2e8ceb7aac4' of type 'Downtime'.
[2021-01-08 15:19:54 +0100] information/Downtime: Added downtime 'testswitch4!ping4!36b1ad18-c5b6-4654-92d8-e2e8ceb7aac4' between '2021-01-08 15:19:54' and '2026-01-08 15:19:54', author: 'API_USER', fixed
[2021-01-08 15:19:54 +0100] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime (from [x.x.x.x]:39664), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:19:54 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39664)
[2021-01-08 15:20:13 +0100] information/ApiListener: New client connection from [x.x.x.x]:39764 (no client certificate)
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39764), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39764)
[2021-01-08 15:20:13 +0100] information/ApiListener: New client connection from [x.x.x.x]:39766 (no client certificate)
[2021-01-08 15:20:13 +0100] information/Checkable: Checkable 'testswitch4' has 2 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!40ffbde6-a9e6-4c56-96ca-5f57c481efd4' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!570a0584-d3aa-4f1d-be48-ccd58b484396' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!e3647339-27a8-40d9-bb91-b0333d31e9f9' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME_HOST' of type 'Notification'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!36b1ad18-c5b6-4654-92d8-e2e8ceb7aac4' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!e3647339-27a8-40d9-bb91-b0333d31e9f9' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!36b1ad18-c5b6-4654-92d8-e2e8ceb7aac4' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: Request: DELETE /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39766), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39766)
[2021-01-08 15:20:13 +0100] information/ApiListener: New client connection from [x.x.x.x]:39768 (no client certificate)
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: Request: PUT /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39768), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39768)
[2021-01-08 15:20:13 +0100] information/ApiListener: New client connection from [x.x.x.x]:39774 (no client certificate)
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:39774), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39774)
[2021-01-08 15:20:13 +0100] information/ApiListener: New client connection from [x.x.x.x]:39776 (no client certificate)
[2021-01-08 15:20:13 +0100] information/Downtime: Triggering downtime 'testswitch4!2f01940b-fdb0-474c-90f6-be83b02b1787' for checkable 'testswitch4'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!2f01940b-fdb0-474c-90f6-be83b02b1787' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/Downtime: Added downtime 'testswitch4!2f01940b-fdb0-474c-90f6-be83b02b1787' between '2021-01-08 15:20:13' and '2026-01-08 15:20:13', author: 'API_USER', fixed
[2021-01-08 15:20:13 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!7068f618-685c-47ad-bcee-ba5376e3c750' for checkable 'testswitch4!CPU'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!7068f618-685c-47ad-bcee-ba5376e3c750' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/Downtime: Added downtime 'testswitch4!CPU!7068f618-685c-47ad-bcee-ba5376e3c750' between '2021-01-08 15:20:13' and '2026-01-08 15:20:13', author: 'API_USER', fixed
[2021-01-08 15:20:13 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!a5117d7e-7e2a-45c9-8831-2b8d8bec2023' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!a5117d7e-7e2a-45c9-8831-2b8d8bec2023' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/Downtime: Added downtime 'testswitch4!Interface Usage!a5117d7e-7e2a-45c9-8831-2b8d8bec2023' between '2021-01-08 15:20:13' and '2026-01-08 15:20:13', author: 'API_USER', fixed
[2021-01-08 15:20:13 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!4119e657-0858-4a9e-9d8f-359243310ecf' for checkable 'testswitch4!ping4'.
[2021-01-08 15:20:13 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!4119e657-0858-4a9e-9d8f-359243310ecf' of type 'Downtime'.
[2021-01-08 15:20:13 +0100] information/Downtime: Added downtime 'testswitch4!ping4!4119e657-0858-4a9e-9d8f-359243310ecf' between '2021-01-08 15:20:13' and '2026-01-08 15:20:13', author: 'API_USER', fixed
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime (from [x.x.x.x]:39776), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:20:13 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:39776)
[2021-01-08 15:20:13 +0100] information/IdoMysqlConnection: Pending queries: 155 (Input: 19/s; Output: 4/s)
[2021-01-08 15:20:13 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 10.9667/s (658/min 4263/5min 12843/15min);
[2021-01-08 15:20:13 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 1.9/s (114/min 773/5min 2401/15min);
[2021-01-08 15:20:13 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-01-08 15:20:23 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 1.41667/s (85/min 744/5min 2373/15min);
[2021-01-08 15:20:43 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 0.816667/s (49/min 711/5min 2326/15min);
[2021-01-08 15:21:11 +0100] information/JsonRpcConnection: No messages for identity 'QS_MASTER_02' have been received in the last 60 seconds.
[2021-01-08 15:21:11 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:21:11 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:21:13 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:13 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:21:13 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:13 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:21:13 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:21:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:13 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:19 +0100] information/Checkable: Checkable 'QS_MASTER_02!cluster' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:21:19 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!cluster!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:21:19 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!cluster!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!cluster' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:21:23 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:23 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:21:23 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:21:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:23 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:21:33 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:33 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:21:33 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:21:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:33 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:21:43 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:43 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:21:43 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:21:43 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:43 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:51 +0100] information/ApiListener: Replayed 578 messages.
[2021-01-08 15:21:53 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 46, rate: 10.05/s (603/min 3463/5min 12091/15min); empty in 10 seconds
[2021-01-08 15:21:53 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:21:53 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:21:53 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:21:53 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:21:53 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:53 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:21:57 +0100] information/ApiListener: Replayed 578 messages.
[2021-01-08 15:22:03 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 139, rate: 9.81667/s (589/min 3339/5min 11941/15min); empty in 14 seconds
[2021-01-08 15:22:03 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:03 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:22:03 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:03 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:22:03 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:22:03 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:03 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:07 +0100] information/ApiListener: Replayed 578 messages.
[2021-01-08 15:22:13 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 287, rate: 9.05/s (543/min 3172/5min 11791/15min); empty in 19 seconds
[2021-01-08 15:22:13 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:13 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:22:13 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:13 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:22:13 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:22:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:13 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:17 +0100] information/ApiListener: Replayed 578 messages.
[2021-01-08 15:22:23 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 410, rate: 6.03333/s (362/min 2964/5min 11610/15min); empty in 33 seconds
[2021-01-08 15:22:23 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.96667/s (178/min 655/5min 2282/15min);
[2021-01-08 15:22:23 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:22:23 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:23 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:22:23 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:22:23 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:23 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:24 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:24 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:33 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 544, rate: 4.05/s (243/min 2801/5min 11448/15min); empty in 40 seconds
[2021-01-08 15:22:33 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.98333/s (179/min 652/5min 2277/15min);
[2021-01-08 15:22:33 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:22:33 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:33 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:22:33 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:22:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:33 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:43 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.86667/s (172/min 650/5min 2284/15min);
[2021-01-08 15:22:43 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 676, rate: 1.33333/s (80/min 2637/5min 11322/15min); empty in 51 seconds
[2021-01-08 15:22:43 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:22:43 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:43 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:22:43 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:22:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:43 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:53 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.85/s (171/min 658/5min 2288/15min);
[2021-01-08 15:22:53 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 805, rate: 0.0166667/s (1/min 2517/5min 11180/15min); empty in 1 minute and 2 seconds
[2021-01-08 15:22:53 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:22:53 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:22:53 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:22:53 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:22:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:53 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:58 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:22:58 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:58 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:22:58 +0100] information/ApiListener: Replayed 579 messages.
[2021-01-08 15:23:03 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 918, rate: 0.0166667/s (1/min 2358/5min 11028/15min); empty in 1 minute and 21 seconds
[2021-01-08 15:23:03 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:03 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:23:03 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:03 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:23:03 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:23:03 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:03 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:13 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1014, rate: 0.0166667/s (1/min 2221/5min 10905/15min); empty in 1 minute and 45 seconds
[2021-01-08 15:23:13 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.7/s (162/min 649/5min 2283/15min);
[2021-01-08 15:23:13 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:13 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:23:13 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:13 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:23:13 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:23:13 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:13 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:23 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1107, rate: 0.0166667/s (1/min 2119/5min 10751/15min); empty in 1 minute and 59 seconds
[2021-01-08 15:23:23 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:23:23 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:23 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:23:23 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:23:23 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:23 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:32 +0100] information/ApiListener: Replayed 579 messages.
[2021-01-08 15:23:33 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:23:33 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:33 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1244, rate:  0/s (0/min 1972/5min 10624/15min); empty in 1 minute and 30 seconds
[2021-01-08 15:23:33 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:23:33 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:33 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:23:33 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:23:33 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:33 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:43 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.61667/s (157/min 654/5min 2288/15min);
[2021-01-08 15:23:43 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1385, rate:  0/s (0/min 1775/5min 10492/15min); empty in 1 minute and 38 seconds
[2021-01-08 15:23:43 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:23:43 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:43 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:23:43 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:23:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:43 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:53 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.5/s (150/min 658/5min 2285/15min);
[2021-01-08 15:23:53 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1496, rate:  0/s (0/min 1668/5min 10347/15min); empty in 2 minutes and 14 seconds
[2021-01-08 15:23:53 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:23:53 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:23:53 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:23:53 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:23:53 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:53 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:23:56 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-01-08 15:24:03 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.43333/s (146/min 649/5min 2281/15min);
[2021-01-08 15:24:03 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1595, rate:  0/s (0/min 1476/5min 10212/15min); empty in 2 minutes and 41 seconds
[2021-01-08 15:24:03 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:03 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:24:03 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:03 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:24:03 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:24:03 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:03 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:07 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:24:07 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:07 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:07 +0100] information/ApiListener: Replayed 580 messages.
[2021-01-08 15:24:13 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.46667/s (148/min 651/5min 2286/15min);
[2021-01-08 15:24:13 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1695, rate: 0.0166667/s (1/min 1363/5min 10111/15min); empty in 2 minutes and 49 seconds
[2021-01-08 15:24:13 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:13 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:24:13 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:13 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:24:13 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:24:13 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:13 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:23 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.75/s (165/min 650/5min 2281/15min);
[2021-01-08 15:24:23 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1849, rate: 0.0166667/s (1/min 1194/5min 9924/15min); empty in 2 minutes
[2021-01-08 15:24:23 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:24:23 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:23 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:24:23 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:24:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:23 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:33 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 1971, rate: 0.0166667/s (1/min 1087/5min 9807/15min); empty in 2 minutes and 41 seconds
[2021-01-08 15:24:33 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:24:33 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:33 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:24:33 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:24:33 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:33 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:41 +0100] information/ApiListener: Replayed 580 messages.
[2021-01-08 15:24:41 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:24:41 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:41 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:42 +0100] information/ApiListener: Replayed 580 messages.
[2021-01-08 15:24:42 +0100] information/ApiListener: Replayed 580 messages.
[2021-01-08 15:24:42 +0100] information/ApiListener: Replayed 580 messages.
[2021-01-08 15:24:43 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2071, rate: 0.0166667/s (1/min 943/5min 9678/15min); empty in 3 minutes and 27 seconds
[2021-01-08 15:24:43 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.53333/s (152/min 655/5min 2288/15min);
[2021-01-08 15:24:43 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:24:43 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:43 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:24:43 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:24:43 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:43 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:53 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2170, rate: 0.0166667/s (1/min 790/5min 9523/15min); empty in 3 minutes and 39 seconds
[2021-01-08 15:24:53 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.53333/s (152/min 651/5min 2281/15min);
[2021-01-08 15:24:53 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:24:53 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:24:53 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:24:53 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:24:53 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:24:53 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:03 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2256, rate: 0.0166667/s (1/min 748/5min 9402/15min); empty in 4 minutes and 22 seconds
[2021-01-08 15:25:03 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:13 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2360, rate:  0/s (0/min 705/5min 9272/15min); empty in 3 minutes and 46 seconds
[2021-01-08 15:25:13 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.35/s (141/min 682/5min 2273/15min);
[2021-01-08 15:25:13 +0100] critical/ApiListener: Client TLS handshake failed (to [x.x.x.x]:5665): Operation canceled
[2021-01-08 15:25:13 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:15 +0100] information/ApiListener: Replayed 580 messages.
[2021-01-08 15:25:15 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:25:15 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:15 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:15 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:25:16 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:25:23 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-01-08 15:25:23 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2507, rate: 0.0166667/s (1/min 670/5min 9099/15min); empty in 2 minutes and 50 seconds
[2021-01-08 15:25:23 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.36667/s (142/min 707/5min 2272/15min);
[2021-01-08 15:25:23 +0100] information/IdoMysqlConnection: Pending queries: 5 (Input: 8/s; Output: 9/s)
[2021-01-08 15:25:23 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:25:23 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:23 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:25:23 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:25:23 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:23 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:33 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2647, rate: 0.0166667/s (1/min 646/5min 8966/15min); empty in 3 minutes and 9 seconds
[2021-01-08 15:25:33 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.45/s (147/min 735/5min 2281/15min);
[2021-01-08 15:25:33 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:25:33 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:33 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:25:33 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:25:33 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:33 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:43 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2740, rate: 0.0166667/s (1/min 628/5min 8791/15min); empty in 4 minutes and 54 seconds
[2021-01-08 15:25:43 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:25:43 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:43 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:25:43 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:25:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:43 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:50 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:25:50 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:25:51 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:25:51 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:25:51 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:51 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:51 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:25:53 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 2839, rate: 0.0166667/s (1/min 606/5min 8646/15min); empty in 4 minutes and 46 seconds
[2021-01-08 15:25:53 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:25:53 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:25:53 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:25:53 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:25:53 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:25:53 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:25 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:26:26 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:26:26 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:26:26 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:26 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 3198, rate:  0/s (0/min 327/5min 8189/15min); empty in 4 minutes and 51 seconds
[2021-01-08 15:26:26 +0100] information/IdoMysqlConnection: Pending queries: 90 (Input: 5/s; Output: 2/s)
[2021-01-08 15:26:26 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:26:26 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:26:26 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:26:26 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:26:26 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:26:26 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:26 +0100] information/ApiListener: Replayed 581 messages.
[2021-01-08 15:26:59 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:26:59 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:59 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:26:59 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 3629, rate:  0/s (0/min 3/5min 7733/15min); empty in 4 minutes and 42 seconds
[2021-01-08 15:26:59 +0100] information/IdoMysqlConnection: Pending queries: 124 (Input: 7/s; Output: 3/s)
[2021-01-08 15:26:59 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:27:09 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 3822, rate:  0/s (0/min 3/5min 7573/15min); empty in 3 minutes and 17 seconds
[2021-01-08 15:27:09 +0100] critical/ApiListener: Client TLS handshake failed (to [x.x.x.x]:5665): Operation canceled
[2021-01-08 15:27:09 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:27:19 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 3962, rate:  0/s (0/min 3/5min 7418/15min); empty in 4 minutes and 42 seconds
[2021-01-08 15:27:19 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:27:19 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:27:19 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:27:19 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:27:19 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:27:19 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:19 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:27:33 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:33 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 4116, rate:  0/s (0/min 2/5min 7155/15min); empty in 6 minutes and 19 seconds
[2021-01-08 15:27:33 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.98333/s (179/min 791/5min 2257/15min);
[2021-01-08 15:27:33 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:27:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:27:33 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:27:33 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:27:33 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:27:33 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:27:33 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:28:07 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:07 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 4528, rate:  0/s (0/min 2/5min 6713/15min); empty in 6 minutes and 10 seconds
[2021-01-08 15:28:07 +0100] information/IdoMysqlConnection: Pending queries: 118 (Input: 8/s; Output: 4/s)
[2021-01-08 15:28:07 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:28:07 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:28:07 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:28:07 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:28:07 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:28:07 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:07 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:28:41 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:41 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 4836, rate: 0.0166667/s (1/min 3/5min 6226/15min); empty in 8 minutes and 53 seconds
[2021-01-08 15:28:41 +0100] information/IdoMysqlConnection: Pending queries: 85 (Input: 4/s; Output: 2/s)
[2021-01-08 15:28:41 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:28:41 +0100] information/Checkable: Checkable 'QS_MASTER_02!Network' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:28:41 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!Network!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:28:41 +0100] information/Checkable: Notifications are disabled for checkable 'QS_MASTER_02!Uptime'.
[2021-01-08 15:28:41 +0100] information/Checkable: Checkable 'QS_MASTER_02!CPU-Usage' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:28:41 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!CPU-Usage!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:28:41 +0100] information/Checkable: Checkable 'QS_MASTER_02!ido' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:28:41 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!ido!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:28:41 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!Network!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!Network' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:28:41 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!CPU-Usage!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!CPU-Usage' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:28:41 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!ido!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!ido' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:28:41 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:28:41 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:28:41 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:41 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:28:41 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:28:41 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:28:41 +0100] information/ApiListener: Replayed 582 messages.
[2021-01-08 15:28:41 +0100] information/ApiListener: Replayed 582 messages.
[2021-01-08 15:29:15 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:29:15 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:15 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 5264, rate: 0.0166667/s (1/min 2/5min 5765/15min); empty in 6 minutes and 59 seconds
[2021-01-08 15:29:15 +0100] information/IdoMysqlConnection: Pending queries: 123 (Input: 4/s; Output: 2/s)
[2021-01-08 15:29:15 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:29:15 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-01-08 15:29:15 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:29:15 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:29:15 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:29:15 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:29:15 +0100] information/Checkable: Checkable 'QS_MASTER_02!Disks' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:29:15 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!Disks!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:29:15 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:15 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:29:49 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:49 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!Disks!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!Disks' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:29:49 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 5667, rate:  0/s (0/min 2/5min 5322/15min); empty in 7 minutes and 51 seconds
[2021-01-08 15:29:49 +0100] information/IdoMysqlConnection: Pending queries: 117 (Input: 8/s; Output: 3/s)
[2021-01-08 15:29:49 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 2, rate: 2.41667/s (145/min 761/5min 2238/15min);
[2021-01-08 15:29:49 +0100] information/Checkable: Checkable 'QS_MASTER_02!npcd' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:29:49 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!npcd!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:29:49 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:29:49 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!npcd!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!npcd' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:29:49 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:29:49 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:29:49 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:29:49 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:29:49 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:29:49 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Replayed 582 messages.
[2021-01-08 15:30:23 +0100] information/ApiListener: Replayed 582 messages.
[2021-01-08 15:30:23 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:30:23 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:23 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-01-08 15:30:23 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 6019, rate:  0/s (0/min 1/5min 4791/15min); empty in 9 minutes and 53 seconds
[2021-01-08 15:30:23 +0100] information/IdoMysqlConnection: Pending queries: 111 (Input: 6/s; Output: 2/s)
[2021-01-08 15:30:23 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 0, rate: 2.38333/s (143/min 761/5min 2217/15min);
[2021-01-08 15:30:23 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:30:23 +0100] information/Checkable: Checkable 'QS_MASTER_02!load' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:30:23 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!load!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:30:23 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!load!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!load' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:30:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:30:23 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:30:23 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:30:23 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:30:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:23 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:25 +0100] information/ApiListener: Replayed 582 messages.
[2021-01-08 15:30:57 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:30:57 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:57 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 6432, rate:  0/s (0/min 1/5min 4319/15min); empty in 8 minutes and 43 seconds
[2021-01-08 15:30:57 +0100] information/IdoMysqlConnection: Pending queries: 111 (Input: 6/s; Output: 3/s)
[2021-01-08 15:30:57 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.16667/s (130/min 768/5min 2214/15min);
[2021-01-08 15:30:57 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:30:57 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:30:57 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:30:57 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:30:57 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:30:57 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:30:57 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:31:31 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:31:31 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 6788, rate:  0/s (0/min 1/5min 3827/15min); empty in 10 minutes and 47 seconds
[2021-01-08 15:31:31 +0100] information/IdoMysqlConnection: Pending queries: 100 (Input: 7/s; Output: 3/s)
[2021-01-08 15:31:31 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:31:31 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:31:31 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:31:31 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:31:31 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:31:31 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:31:31 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:32:05 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:05 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 7145, rate:  0/s (0/min 1/5min 3325/15min); empty in 11 minutes and 11 seconds
[2021-01-08 15:32:05 +0100] information/IdoMysqlConnection: Pending queries: 91 (Input: 2/s; Output: 1/s)
[2021-01-08 15:32:05 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:32:05 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:32:05 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:32:05 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:32:05 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:32:05 +0100] information/Checkable: Checkable 'QS_MASTER_02!memory' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:32:05 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!memory!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:32:05 +0100] information/Checkable: Checkable 'QS_MASTER_02!icinga' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-01-08 15:32:05 +0100] information/Notification: Sending 'Problem' notification 'QS_MASTER_02!icinga!NOTIFICATION_NAME_SERVICE' for user 'ICINGA_USERNAME'
[2021-01-08 15:32:05 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:05 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:39 +0100] information/ApiListener: Replayed 582 messages.
[2021-01-08 15:32:40 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:32:40 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:40 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!memory!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!memory' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:32:40 +0100] information/Notification: Completed sending 'Problem' notification 'QS_MASTER_02!icinga!NOTIFICATION_NAME_SERVICE' for checkable 'QS_MASTER_02!icinga' and user 'ICINGA_USERNAME' using command 'ICINGA_NAME'.
[2021-01-08 15:32:40 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 7700, rate:  0/s (0/min 1/5min 2697/15min); empty in 8 minutes and 14 seconds
[2021-01-08 15:32:40 +0100] information/IdoMysqlConnection: Pending queries: 158 (Input: 10/s; Output: 4/s)
[2021-01-08 15:32:40 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 1, rate: 2.71667/s (163/min 743/5min 2191/15min);
[2021-01-08 15:32:40 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:32:40 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-08 15:32:40 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-08 15:32:40 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-08 15:32:40 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-08 15:32:40 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:32:40 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:33:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:40052 (no client certificate)
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:40052), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:40052)
[2021-01-08 15:33:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:40054 (no client certificate)
[2021-01-08 15:33:11 +0100] information/Checkable: Checkable 'testswitch4' has 2 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:33:11 +0100] information/Notification: Sending 'DowntimeEnd' notification 'testswitch4!DOWNTIME_NAME' for user 'ICINGA_USERNAME'
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!2f01940b-fdb0-474c-90f6-be83b02b1787' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!a5117d7e-7e2a-45c9-8831-2b8d8bec2023' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:33:11 +0100] information/Checkable: Checkable 'testswitch4!CPU' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!7068f618-685c-47ad-bcee-ba5376e3c750' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!7068f618-685c-47ad-bcee-ba5376e3c750' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!DOWNTIME_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME_HOST' of type 'Notification'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!4119e657-0858-4a9e-9d8f-359243310ecf' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!a5117d7e-7e2a-45c9-8831-2b8d8bec2023' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: Request: DELETE /v1/objects/hosts/testswitch4 (from [x.x.x.x]:40054), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:40054)
[2021-01-08 15:33:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:40056 (no client certificate)
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: Request: PUT /v1/objects/hosts/testswitch4 (from [x.x.x.x]:40056), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:40056)
[2021-01-08 15:33:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:40062 (no client certificate)
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: Request: GET /v1/objects/hosts/testswitch4 (from [x.x.x.x]:40062), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:40062)
[2021-01-08 15:33:11 +0100] information/ApiListener: New client connection from [x.x.x.x]:40064 (no client certificate)
[2021-01-08 15:33:11 +0100] information/Downtime: Triggering downtime 'testswitch4!6540a4c9-fbe8-4789-b47a-a8c6fa9d5fdc' for checkable 'testswitch4'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!6540a4c9-fbe8-4789-b47a-a8c6fa9d5fdc' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/Downtime: Added downtime 'testswitch4!6540a4c9-fbe8-4789-b47a-a8c6fa9d5fdc' between '2021-01-08 15:33:11' and '2026-01-08 15:33:11', author: 'API_USER', fixed
[2021-01-08 15:33:11 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!dfed6583-cb63-4cd6-8548-74d3ad9f81bb' for checkable 'testswitch4!CPU'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!dfed6583-cb63-4cd6-8548-74d3ad9f81bb' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/Downtime: Added downtime 'testswitch4!CPU!dfed6583-cb63-4cd6-8548-74d3ad9f81bb' between '2021-01-08 15:33:11' and '2026-01-08 15:33:11', author: 'API_USER', fixed
[2021-01-08 15:33:11 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!9f773080-da82-44d0-8448-379e42679e26' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!9f773080-da82-44d0-8448-379e42679e26' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/Downtime: Added downtime 'testswitch4!Interface Usage!9f773080-da82-44d0-8448-379e42679e26' between '2021-01-08 15:33:11' and '2026-01-08 15:33:11', author: 'API_USER', fixed
[2021-01-08 15:33:11 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!7b7eec11-39ed-4736-ab26-402de5a57d64' for checkable 'testswitch4!ping4'.
[2021-01-08 15:33:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!7b7eec11-39ed-4736-ab26-402de5a57d64' of type 'Downtime'.
[2021-01-08 15:33:11 +0100] information/Downtime: Added downtime 'testswitch4!ping4!7b7eec11-39ed-4736-ab26-402de5a57d64' between '2021-01-08 15:33:11' and '2026-01-08 15:33:11', author: 'API_USER', fixed
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime (from [x.x.x.x]:40064), user: ICINGA_USERNAME, agent: Mojolicious (Perl), status: OK).
[2021-01-08 15:33:11 +0100] information/HttpServerConnection: HTTP client disconnected (from [x.x.x.x]:40064)
[2021-01-08 15:33:14 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-08 15:33:14 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-08 15:33:14 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
<segfault>

QS_MASTER_01 - /var/log/messages

Jan  8 15:33:14 QS_MASTER_01 kernel: icinga2[3794]: segfault at fffffffffffffff8 ip 00007fc5575e6f7b sp 00007fc551ab2430 error 5 in libstdc++.so.6.0.19[7fc557528000+e9000]
Jan  8 15:33:14 QS_MASTER_01 kernel: [148908.036718] icinga2[3794]: segfault at fffffffffffffff8 ip 00007fc5575e6f7b sp 00007fc551ab2430 error 5 in libstdc++.so.6.0.19[7fc557528000+e9000]
Jan  8 15:33:15 QS_MASTER_01 systemd: icinga2.service: main process exited, code=exited, status=139/n/a
Jan  8 15:33:15 QS_MASTER_01 systemd: Unit icinga2.service entered failed state.
Jan  8 15:33:15 QS_MASTER_01 systemd: icinga2.service failed.

QS_MASTER_02 - /var/log/icinga2/icinga2.log

[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:16:11 +0100] information/Downtime: Triggering downtime 'testswitch4!1427520e-2c49-4f47-9a84-3754cf6dd7b5' for checkable 'testswitch4'.
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!1427520e-2c49-4f47-9a84-3754cf6dd7b5' of type 'Downtime'.
[2021-01-08 15:16:11 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!cb3de749-46d8-48ba-89d1-39923eb27e0e' for checkable 'testswitch4!CPU'.
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!cb3de749-46d8-48ba-89d1-39923eb27e0e' of type 'Downtime'.
[2021-01-08 15:16:11 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!e4ffd2b2-1679-490f-958f-7f2769b2a22d' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!e4ffd2b2-1679-490f-958f-7f2769b2a22d' of type 'Downtime'.
[2021-01-08 15:16:11 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!9173bd81-4fbc-43e9-b624-3635010ad719' for checkable 'testswitch4!ping4'.
[2021-01-08 15:16:11 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!9173bd81-4fbc-43e9-b624-3635010ad719' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Checkable: Checkable 'testswitch4!Interface Usage' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!e4ffd2b2-1679-490f-958f-7f2769b2a22d' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Checkable: Checkable 'testswitch4!CPU' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!cb3de749-46d8-48ba-89d1-39923eb27e0e' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Checkable: Checkable 'testswitch4' has 2 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:17:22 +0100] information/Notification: Sending 'DowntimeEnd' notification 'testswitch4!DOWNTIME_NAME' for user 'ICINGA_USERNAME'
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!1427520e-2c49-4f47-9a84-3754cf6dd7b5' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Checkable: Checkable 'testswitch4!ping4' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!9173bd81-4fbc-43e9-b624-3635010ad719' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Notification: Completed sending 'DowntimeEnd' notification 'testswitch4!DOWNTIME_NAME' for checkable 'testswitch4' and user 'ICINGA_USERNAME' using command 'NOTIFICATION_NAME'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!NOTIFICATION_NAME_HOST' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:17:22 +0100] information/Downtime: Triggering downtime 'testswitch4!14300750-5d1f-43ce-89bf-8592ef6afca3' for checkable 'testswitch4'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!14300750-5d1f-43ce-89bf-8592ef6afca3' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!d9ed52fc-d35f-4c09-aa12-f6fff1b80987' for checkable 'testswitch4!CPU'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!d9ed52fc-d35f-4c09-aa12-f6fff1b80987' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!6156e0f1-621a-4df7-ba86-320cfd8113e5' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!6156e0f1-621a-4df7-ba86-320cfd8113e5' of type 'Downtime'.
[2021-01-08 15:17:22 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!dc9bad9b-27c8-40c9-bcb9-8f0ae6dac741' for checkable 'testswitch4!ping4'.
[2021-01-08 15:17:22 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!dc9bad9b-27c8-40c9-bcb9-8f0ae6dac741' of type 'Downtime'.
[2021-01-08 15:17:31 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2021-01-08 15:18:03 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-01-08 15:18:03 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 0, rate: 2.96667/s (178/min 822/5min 2450/15min);
[2021-01-08 15:18:03 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 14.3833/s (863/min 3825/5min 11217/15min);
[2021-01-08 15:18:04 +0100] information/IdoMysqlConnection: Pending queries: 1 (Input: 7/s; Output: 7/s)
[2021-01-08 15:18:35 +0100] information/Checkable: Checkable 'testswitch4!CPU' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!d9ed52fc-d35f-4c09-aa12-f6fff1b80987' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/Checkable: Checkable 'testswitch4!Interface Usage' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!6156e0f1-621a-4df7-ba86-320cfd8113e5' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/Checkable: Checkable 'testswitch4' has 2 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:35 +0100] information/Notification: Sending 'DowntimeEnd' notification 'testswitch4!DOWNTIME_NAME' for user 'ICINGA_USERNAME'
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!14300750-5d1f-43ce-89bf-8592ef6afca3' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/Checkable: Checkable 'testswitch4!ping4' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!dc9bad9b-27c8-40c9-bcb9-8f0ae6dac741' of type 'Downtime'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/Notification: Completed sending 'DowntimeEnd' notification 'testswitch4!DOWNTIME_NAME' for checkable 'testswitch4' and user 'ICINGA_USERNAME' using command 'NOTIFICATION_NAME'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!NOTIFICATION_NAME_HOST' of type 'Notification'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:18:35 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:18:36 +0100] information/Downtime: Triggering downtime 'testswitch4!9716a401-1d6f-4961-958b-c3bcbe426384' for checkable 'testswitch4'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!9716a401-1d6f-4961-958b-c3bcbe426384' of type 'Downtime'.
[2021-01-08 15:18:36 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!54cc4fc1-a911-4891-a8b5-0e7ea13d1fe0' for checkable 'testswitch4!CPU'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!54cc4fc1-a911-4891-a8b5-0e7ea13d1fe0' of type 'Downtime'.
[2021-01-08 15:18:36 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!0997428b-730c-4a0f-8038-208bc84f5bbe' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!0997428b-730c-4a0f-8038-208bc84f5bbe' of type 'Downtime'.
[2021-01-08 15:18:36 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!ac3ab9c8-ad76-4347-aaf2-dc87930f9082' for checkable 'testswitch4!ping4'.
[2021-01-08 15:18:36 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!ac3ab9c8-ad76-4347-aaf2-dc87930f9082' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Checkable: Checkable 'testswitch4!Interface Usage' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!0997428b-730c-4a0f-8038-208bc84f5bbe' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Checkable: Checkable 'testswitch4!CPU' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!54cc4fc1-a911-4891-a8b5-0e7ea13d1fe0' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Checkable: Checkable 'testswitch4' has 2 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:54 +0100] information/Notification: Sending 'DowntimeEnd' notification 'testswitch4!DOWNTIME_NAME' for user 'ICINGA_USERNAME'
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!9716a401-1d6f-4961-958b-c3bcbe426384' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Checkable: Checkable 'testswitch4!ping4' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!ac3ab9c8-ad76-4347-aaf2-dc87930f9082' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/Notification: Completed sending 'DowntimeEnd' notification 'testswitch4!DOWNTIME_NAME' for checkable 'testswitch4' and user 'ICINGA_USERNAME' using command 'NOTIFICATION_NAME'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!NOTIFICATION_NAME_HOST' of type 'Notification'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4' of type 'Host'.
[2021-01-08 15:18:54 +0100] information/Downtime: Triggering downtime 'testswitch4!3a24d944-e233-43fd-868f-18bac207ad68' for checkable 'testswitch4'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!3a24d944-e233-43fd-868f-18bac207ad68' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Downtime: Triggering downtime 'testswitch4!CPU!d20edbd0-172b-4a45-8630-6c8bdde73819' for checkable 'testswitch4!CPU'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!CPU!d20edbd0-172b-4a45-8630-6c8bdde73819' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Downtime: Triggering downtime 'testswitch4!Interface Usage!fe40edf5-b121-4db7-97af-8be820ca7268' for checkable 'testswitch4!Interface Usage'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!Interface Usage!fe40edf5-b121-4db7-97af-8be820ca7268' of type 'Downtime'.
[2021-01-08 15:18:54 +0100] information/Downtime: Triggering downtime 'testswitch4!ping4!32e363f2-6dc7-4017-ac88-6474d06e6769' for checkable 'testswitch4!ping4'.
[2021-01-08 15:18:54 +0100] information/ConfigObjectUtility: Created and activated object 'testswitch4!ping4!32e363f2-6dc7-4017-ac88-6474d06e6769' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/Checkable: Checkable 'testswitch4!CPU' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!d20edbd0-172b-4a45-8630-6c8bdde73819' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/Checkable: Checkable 'testswitch4!Interface Usage' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!fe40edf5-b121-4db7-97af-8be820ca7268' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/Checkable: Checkable 'testswitch4!ping4' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!32e363f2-6dc7-4017-ac88-6474d06e6769' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/Checkable: Checkable 'testswitch4' has 2 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-01-08 15:19:46 +0100] information/Notification: Sending 'DowntimeEnd' notification 'testswitch4!DOWNTIME_NAME' for user 'ICINGA_USERNAME'
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!3a24d944-e233-43fd-868f-18bac207ad68' of type 'Downtime'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage' of type 'Service'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU' of type 'Service'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!Interface Usage!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!CPU!NOTIFICATION_NAME_SERVICE' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!NOTIFICATION_NAME_HOST' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!DOWNTIME_NAME' of type 'Notification'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4!ping4' of type 'Service'.
[2021-01-08 15:19:46 +0100] information/ConfigObjectUtility: Deleted object 'testswitch4' of type 'Host'.
[2021-01-08 15:19:46 +0100] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in '/var/log/icinga2/crash/report.1610115586.361180'

[2021-01-08 15:19:53 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-01-08 15:21:13 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33342
[2021-01-08 15:21:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33364
[2021-01-08 15:21:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33378
[2021-01-08 15:21:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33384
[2021-01-08 15:21:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33388
[2021-01-08 15:22:03 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33394
[2021-01-08 15:22:13 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33404
[2021-01-08 15:22:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33434
[2021-01-08 15:22:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33446
[2021-01-08 15:22:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33452
[2021-01-08 15:22:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33458
[2021-01-08 15:23:03 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33462
[2021-01-08 15:23:13 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33466
[2021-01-08 15:23:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33496
[2021-01-08 15:23:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33502
[2021-01-08 15:23:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33508
[2021-01-08 15:23:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33514
[2021-01-08 15:24:03 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33518
[2021-01-08 15:24:13 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33524
[2021-01-08 15:24:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33536
[2021-01-08 15:24:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33542
[2021-01-08 15:24:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33546
[2021-01-08 15:24:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33552
[2021-01-08 15:25:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33568
[2021-01-08 15:25:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33574
[2021-01-08 15:25:43 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33580
[2021-01-08 15:25:53 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33588
[2021-01-08 15:26:26 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33598
[2021-01-08 15:27:19 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33660
[2021-01-08 15:27:33 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33666
[2021-01-08 15:28:07 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33680
[2021-01-08 15:28:41 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33700
[2021-01-08 15:29:15 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33720
[2021-01-08 15:29:49 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33742
[2021-01-08 15:30:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33754
[2021-01-08 15:30:57 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33770
[2021-01-08 15:31:31 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33788
[2021-01-08 15:32:05 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33802
[2021-01-08 15:32:40 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33850
[2021-01-08 15:33:15 +0100] critical/ApiListener: Client TLS handshake failed (from [x.x.x.x]:33862): Connection reset by peer
<no log messages after this until service restart>

QS_MASTER_02 - /var/log/icinga2/crash/report.1610115586.361180

  Application version: 2.11.7-1

System information:
  Platform: Red Hat Enterprise Linux Server
  Platform version: 7.9 (Maipo)
  Kernel: Linux
  Kernel version: 3.10.0-1160.11.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-322-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

        (0) libc.so.6: gsignal (+0x37) [0x7f4f0ee02387]
        (1) libc.so.6: abort (+0x148) [0x7f4f0ee03a78]
        (2) libc.so.6: <unknown function> (+0x78ed7) [0x7f4f0ee44ed7]
        (3) libc.so.6: <unknown function> (+0x7f3e4) [0x7f4f0ee4b3e4]
        (4) libc.so.6: <unknown function> (+0x814db) [0x7f4f0ee4d4db]
        (5) icinga2: icinga::Notification::ExecuteNotificationHelper(icinga::NotificationType, boost::intrusive_ptr<icinga::User> const&, boost::intrusive_ptr<icinga::CheckResult> const&, bool, icinga::String const&, icinga::String const&) (+0x1df) [0xa31e7f]
        (6) icinga2: std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (icinga::Notification::*)(icinga::NotificationType, boost::intrusive_ptr<icinga::User> const&, boost::intrusive_ptr<icinga::CheckResult> const&, bool, icinga::String const&, icinga::String const&)> (icinga::Notification*, icinga::NotificationType, boost::intrusive_ptr<icinga::User>, boost::intrusive_ptr<icinga::CheckResult>, bool, icinga::String, icinga::String)> >::_M_invoke(std::_Any_data const&) (+0x42) [0xa2bf22]
        (7) /usr/lib64/icinga2/sbin/icinga2() [0x907b33]
        (8) icinga2: boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<bool icinga::ThreadPool::Post<std::function<void ()> >(std::function<void ()>, icinga::SchedulerPolicy)::{lambda()#1}>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, std::allocator<void>*, boost::system::error_code const&, unsigned long) (+0x106) [0xbfed66]
        (9) /usr/lib64/icinga2/sbin/icinga2() [0x6313e1]
        (10) /usr/lib64/icinga2/sbin/icinga2() [0x631732]
        (11) icinga2: boost_asio_detail_posix_thread_function (+0xf) [0x823d5f]
        (12) libpthread.so.0: <unknown function> (+0x7ea5) [0x7f4f0f1a1ea5]
        (13) libc.so.6: clone (+0x6d) [0x7f4f0eeca96d]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

QS_MASTER_02 - /var/log/icinga2/error.log

*** Error in `/usr/lib64/icinga2/sbin/icinga2': corrupted size vs. prev_size: 0x00007f4e2c015860 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f3e4)[0x7f4f0ee4b3e4]
/lib64/libc.so.6(+0x814db)[0x7f4f0ee4d4db]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga12Notification25ExecuteNotificationHelperENS_16NotificationTypeERKN5boost13intrusive_ptrINS_4UserEEERKNS3_INS_11CheckResultEEEbRKNS_6StringESE_+0x1df)[0xa31e7f]
/usr/lib64/icinga2/sbin/icinga2(_ZNSt17_Function_handlerIFvvESt5_BindIFSt7_Mem_fnIMN6icinga12NotificationEFvNS3_16NotificationTypeERKN5boost13intrusive_ptrINS3_4UserEEERKNS7_INS3_11CheckResultEEEbRKNS3_6StringESI_EEPS4_S5_S9_SD_bSG_SG_EEE9_M_invokeERKSt9_Any_data+0x42)[0xa2bf22]
/usr/lib64/icinga2/sbin/icinga2[0x907b33]
/usr/lib64/icinga2/sbin/icinga2(_ZN5boost4asio6detail11executor_opINS1_15work_dispatcherIZN6icinga10ThreadPool4PostISt8functionIFvvEEEEbT_NS4_15SchedulerPolicyEEUlvE_EESaIvENS1_19scheduler_operationEE11do_completeEPvPSF_RKNS_6system10error_codeEm+0x106)[0xbfed66]
/usr/lib64/icinga2/sbin/icinga2[0x6313e1]
/usr/lib64/icinga2/sbin/icinga2[0x631732]
/usr/lib64/icinga2/sbin/icinga2(boost_asio_detail_posix_thread_function+0xf)[0x823d5f]
/lib64/libpthread.so.0(+0x7ea5)[0x7f4f0f1a1ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f4f0eeca96d]
======= Memory map: ========
<...>
Caught SIGABRT.
Current time: 2021-01-08 15:19:46 +010
K0nne commented 3 years ago

Today in the morning the icinga2 process on the primary master was in "failed" state, while the 2nd one was still "active (running)", so I just started the first one again, At this point I wondered how the 2nd one had survived. After this in IcingaWeb2 the "backend not available" flapping from #8587 started again. Those 2 crashes are connected.

The 2nd master somehow survived (?!) the crash, still accepting connections, but nothing more. This icinga was still writing in an already rotated icinga2.log

QS_MASTER_02 - /var/log/icinga2/icinga2.log

[2021-01-08 15:30:23 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33754
[2021-01-08 15:30:57 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33770
[2021-01-08 15:31:31 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33788
[2021-01-08 15:32:05 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33802
[2021-01-08 15:32:40 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:33850
[2021-01-08 15:33:15 +0100] critical/ApiListener: Client TLS handshake failed (from [x.x.x.x]:33862): Connection reset by peer
<"systemctl start icinga2" @QS_MASTER_01>
[2021-01-11 09:54:40 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:36128
[2021-01-11 09:54:50 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:36132
[2021-01-11 09:55:00 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:36142
[2021-01-11 09:55:30 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:36212
...
[2021-01-11 13:58:31 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:44256
[2021-01-11 13:59:28 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:44302
[2021-01-11 14:00:16 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:44352
[2021-01-11 14:01:02 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:44402
[2021-01-11 14:01:49 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_01' from [x.x.x.x]:44442

QS_MASTER_02 - sudo systemctl status icinga2 active (running) while already dead

 sudo systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-01-07 22:04:40 CET; 3 days ago
  Process: 3616 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 3660 (icinga2)
    Tasks: 73
   Memory: 66.1M
   CGroup: /system.slice/icinga2.service
           ├─3660 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /data/icinga/log/error.log
           ├─3778 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /data/icinga/log/error.log
           └─3780 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /data/icinga/log/error.log

QS_MASTER_01 went in a sync-loop after this since 9:54 this morning, writing only those blocks after the restart.

QS_MASTER_01 - /var/log/icinga2/icinga2.log

[2021-01-11 14:06:22 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-11 14:06:22 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:06:22 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-01-11 14:06:22 +0100] information/WorkQueue: #7 (GraphiteWriter, graphite) items: 0, rate: 2.55/s (153/min 697/5min 2186/15min);
[2021-01-11 14:06:22 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 100803, rate:  0/s (0/min 0/5min 33/15min); empty in 2 hours, 17 minutes and 4 seconds
[2021-01-11 14:06:22 +0100] information/IdoMysqlConnection: Pending queries: 139 (Input: 7/s; Output: 3/s)
[2021-01-11 14:06:22 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-11 14:06:22 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-01-11 14:06:22 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-11 14:06:22 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-11 14:06:22 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-11 14:06:22 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-11 14:06:22 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:06:22 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.

[2021-01-11 14:07:06 +0100] information/ApiListener: Replayed 1 messages.
[2021-01-11 14:07:06 +0100] information/ApiListener: Finished sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Finished syncing endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:07:06 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 101386, rate:  0/s (0/min 0/5min 33/15min); empty in 2 hours, 7 minutes and 36 seconds
[2021-01-11 14:07:06 +0100] information/IdoMysqlConnection: Pending queries: 158 (Input: 7/s; Output: 3/s)
[2021-01-11 14:07:06 +0100] information/ApiListener: Reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-11 14:07:06 +0100] information/ApiListener: New client connection for identity 'QS_MASTER_02' to [x.x.x.x]:5665
[2021-01-11 14:07:06 +0100] information/ApiListener: Sending config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Finished reconnecting to endpoint 'QS_MASTER_02' via host 'x.x.x.x' and port '5665'
[2021-01-11 14:07:06 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_01' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Syncing configuration files for global zone 'linux-commands' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Syncing configuration files for global zone 'windows-commands' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] warning/JsonRpcConnection: API client disconnected for identity 'QS_MASTER_02'
[2021-01-11 14:07:06 +0100] warning/ApiListener: Removing API client for endpoint 'QS_MASTER_02'. 0 API clients left.
[2021-01-11 14:07:06 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_02' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Syncing configuration files for zone 'ZONE_03' to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Finished sending config file updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'QS_MASTER_02'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'QS_MASTER_02' in zone 'master'.
[2021-01-11 14:07:06 +0100] information/ApiListener: Sending replay log for endpoint 'QS_MASTER_02' in zone 'master'.

sudo systemctl restart icinga2 @QS_MASTER_02 took almost 2 minutes.

After this the 2nd master is working again and wrote in a new icinga2.log. I see a lot of recovery and notifications @QS_MASTER_02.