search

redhat-cop / keepalived-operator

An operator to manage VIPs backed by keepalived
Apache License 2.0
117 stars 36 forks source link

Keepalived VIP conflict #78

Closed osirisstar-hub closed 2 years ago

osirisstar-hub commented 2 years ago

I have a daemon that keeps trying to setup a VIP on a node when that VIP already exists on another node.

Fri Sep 17 16:07:21 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 16:07:21 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 16:07:21 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 16:07:21 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117

This results in kube-proxy failures such as this: can't open "externalIP for deepa-zrr/backoffice-ip-elm" (10.22.4.117:8010/tcp), skipping this externalIP: listen tcp4 10.22.4.117:8010: bind: cannot assign requested address

I am having this issue across the entire cluster where daemons keep trying to take over a VIP when it is functional and active.

raffaelespazzoli commented 2 years ago

the first part of this issue is clear to me. A possible reason why multiple nodes are trying to claim the same VIP is if there is no L2 connectivity between the nodes. The second part of the description is not clear, when does that error message appears? If the VIP was created on that node, why would kube-proxy error out that way? Can you provide more context on where/when you see that error message?

osirisstar-hub commented 2 years ago

the VIP is already available on another node so there is a conflict when another node tries to assign the IP address to itself.

raffaelespazzoli commented 2 years ago

that is not what the message seems to be saying, kube-proxy seems to be saying that it cannot bind to that address. I think kube-proxy is a daemon with local visibility, it does not know if other kube-proxy have bound to the same VIP.

osirisstar-hub commented 2 years ago

I think it would try to assign an IP that's already in use and fail. I can ping the VIP from that VM with no problem.

sh-4.4# ping 10.22.4.117 PING 10.22.4.117 (10.22.4.117) 56(84) bytes of data. 64 bytes from 10.22.4.117: icmp_seq=1 ttl=64 time=1.01 ms 64 bytes from 10.22.4.117: icmp_seq=2 ttl=64 time=0.102 ms 64 bytes from 10.22.4.117: icmp_seq=3 ttl=64 time=0.130 ms 64 bytes from 10.22.4.117: icmp_seq=4 ttl=64 time=0.103 ms 64 bytes from 10.22.4.117: icmp_seq=5 ttl=64 time=0.950 ms

osirisstar-hub commented 2 years ago

So not sure why the VM is trying to assign a VIP that is already in use and it can ping.

raffaelespazzoli commented 2 years ago

is there L2 connectivity between the nodes?

osirisstar-hub commented 2 years ago

No problem doing a ping between the nodes. There are all on the same subnet so yes.

What causes advertisement timeouts?

Fri Sep 17 15:25:58 2021: (system-testing/ag-1) Receive advertisement timeout

osirisstar-hub commented 2 years ago

So we should have an ARP entry for the IP address? That kind of L2 connectivity?

raffaelespazzoli commented 2 years ago

no the various keepalived instances use the VRRP (L2 multicast) protocol to communicate with each other and elect the leader which will carry the VIP. If they don't "see" each other, they will all try to carry the VIP.

osirisstar-hub commented 2 years ago

Recommend increasing the timeout? They seem to be able to communicate according to this log but there appears to be a timeout occuring before the advert is picked up:

Fri Sep 17 18:59:21 2021: (system-testing/ag-1) Receive advertisement timeout Fri Sep 17 18:59:21 2021: (system-testing/ag-1) Entering MASTER STATE Fri Sep 17 18:59:21 2021: (system-testing/ag-1) setting VIPs. Fri Sep 17 18:59:21 2021: (system-testing/ag-1) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.104 Fri Sep 17 18:59:21 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 18:59:21 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 18:59:21 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 18:59:21 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 18:59:21 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 18:59:22 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 18:59:22 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 18:59:22 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 18:59:22 2021: (system-testing/ag-1) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 18:59:22 2021: (system-testing/ag-1) Entering BACKUP STATE Fri Sep 17 18:59:22 2021: (system-testing/ag-1) removing VIPs. Fri Sep 17 19:00:07 2021: Interface veth9817c5ae deleted Fri Sep 17 19:00:10 2021: Interface veth7d807c11 added Fri Sep 17 19:00:10 2021: Interface vethf0ad3448 deleted Fri Sep 17 19:00:11 2021: Interface vethf74880da added Fri Sep 17 19:00:19 2021: Interface veth77ec3e71 added Fri Sep 17 19:00:19 2021: Interface veth1b20cbaf added Fri Sep 17 19:00:44 2021: Interface veth57b650cc added Fri Sep 17 19:00:44 2021: Interface vethe61b26e7 added Fri Sep 17 19:00:57 2021: Interface veth1e44de76 added Fri Sep 17 19:01:27 2021: Interface veth4e4ec551 deleted Fri Sep 17 19:01:27 2021: Interface veth536d6b28 added Fri Sep 17 19:01:27 2021: Interface vethdf00a5d5 deleted Fri Sep 17 19:01:30 2021: Interface veth877a3b6b deleted Fri Sep 17 19:01:30 2021: Interface veth25cfb392 deleted Fri Sep 17 19:01:30 2021: Interface vethd0d4d813 deleted Fri Sep 17 19:01:46 2021: Interface vethd2cf46e0 added Fri Sep 17 19:01:49 2021: Interface veth60a1357e added Fri Sep 17 19:02:08 2021: Interface veth39091122 added Fri Sep 17 19:02:19 2021: Interface veth5b001b03 added Fri Sep 17 19:02:25 2021: Interface veth52e160e1 deleted Fri Sep 17 19:02:33 2021: Interface veth7d807c11 deleted Fri Sep 17 19:02:36 2021: Interface veth2970659c added Fri Sep 17 19:02:36 2021: Interface vethcbcb25a2 deleted Fri Sep 17 19:02:37 2021: Interface vethe707fda8 deleted Fri Sep 17 19:03:00 2021: Interface veth34adbd82 added Fri Sep 17 19:03:11 2021: Interface veth0d9182b9 added Fri Sep 17 19:03:13 2021: Interface vethc87fe391 added Fri Sep 17 19:03:20 2021: Interface vetha3e7965a deleted Fri Sep 17 19:03:24 2021: Interface veth050d638a added Fri Sep 17 19:03:40 2021: Interface veth536d6b28 deleted Fri Sep 17 19:03:40 2021: Interface veth60a1357e deleted Fri Sep 17 19:03:42 2021: Interface vethcc3b8e9a deleted Fri Sep 17 19:03:42 2021: Interface veth77ec3e71 deleted Fri Sep 17 19:03:42 2021: Interface veth5b001b03 deleted Fri Sep 17 19:03:42 2021: Interface veth1b20cbaf deleted Fri Sep 17 19:03:43 2021: Interface veth849b82cb added Fri Sep 17 19:03:45 2021: Interface vethd3acb48d added Fri Sep 17 19:04:45 2021: Interface vethe61b26e7 deleted Fri Sep 17 19:05:20 2021: Interface veth2a8f74c8 added Fri Sep 17 19:05:21 2021: Interface vethd67ba93e added Fri Sep 17 19:05:37 2021: Interface veth052b87a0 added Fri Sep 17 19:05:38 2021: Interface vethf3ea3a4f added Fri Sep 17 19:05:51 2021: Interface vethc87fe391 deleted Fri Sep 17 19:05:52 2021: Interface vethb3a3c090 added Fri Sep 17 19:06:16 2021: Interface veth04ec3d63 added Fri Sep 17 19:06:33 2021: Interface vethe2620081 added Fri Sep 17 19:06:51 2021: Interface veth0d9182b9 deleted Fri Sep 17 19:07:04 2021: Interface veth050d638a deleted Fri Sep 17 19:07:07 2021: Interface veth86b3dcc6 added Fri Sep 17 19:07:53 2021: Interface veth34adbd82 deleted Fri Sep 17 19:07:53 2021: Interface veth04ec3d63 deleted Fri Sep 17 19:07:53 2021: Interface vethd2cf46e0 deleted Fri Sep 17 19:08:01 2021: Interface vethd3acb48d deleted Fri Sep 17 19:08:42 2021: Interface veth8f7d6377 added Fri Sep 17 19:08:46 2021: Interface vetha9ba698c added Fri Sep 17 19:08:50 2021: Interface veth449f9063 added Fri Sep 17 19:08:53 2021: Interface vethe2620081 deleted Fri Sep 17 19:08:53 2021: Interface vethb3a3c090 deleted Fri Sep 17 19:08:53 2021: Interface vethf74880da deleted Fri Sep 17 19:08:54 2021: Interface veth2a8f74c8 deleted Fri Sep 17 19:08:54 2021: Interface veth86b3dcc6 deleted Fri Sep 17 19:09:02 2021: (system-testing/ag-1) Receive advertisement timeout Fri Sep 17 19:09:02 2021: (system-testing/ag-1) Entering MASTER STATE Fri Sep 17 19:09:02 2021: (system-testing/ag-1) setting VIPs. Fri Sep 17 19:09:02 2021: (system-testing/ag-1) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.104 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:09:02 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 19:09:02 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 19:09:02 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 19:09:02 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:09:02 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:09:02 2021: (system-testing/ag-1) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:09:02 2021: (system-testing/ag-1) Entering BACKUP STATE Fri Sep 17 19:09:02 2021: (system-testing/ag-1) removing VIPs. Fri Sep 17 19:09:02 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:09:02 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 19:09:02 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 19:09:47 2021: Interface veth1eb11998 added Fri Sep 17 19:09:54 2021: Interface veth2970659c deleted Fri Sep 17 19:09:54 2021: Interface veth052b87a0 deleted Fri Sep 17 19:09:54 2021: Interface vethd67ba93e deleted Fri Sep 17 19:09:54 2021: Interface vethf3ea3a4f deleted Fri Sep 17 19:09:54 2021: Interface veth849b82cb deleted Fri Sep 17 19:10:09 2021: Interface veth30ac20a7 added Fri Sep 17 19:10:23 2021: Interface veth7135e58f added Fri Sep 17 19:10:47 2021: Interface veth9cdaded6 added Fri Sep 17 19:10:54 2021: Interface vethd51de0a8 added Fri Sep 17 19:10:54 2021: Interface veth8f7d6377 deleted Fri Sep 17 19:11:10 2021: Interface veth4926b2b3 added Fri Sep 17 19:11:26 2021: Interface vethc765b21f added Fri Sep 17 19:11:39 2021: Interface veth22b381d1 added Fri Sep 17 19:11:45 2021: Interface vethbccd808b added Fri Sep 17 19:12:11 2021: Interface vethdb837fe2 added Fri Sep 17 19:12:11 2021: Interface veth30ac20a7 deleted Fri Sep 17 19:12:14 2021: Interface veth1eb11998 deleted Fri Sep 17 19:12:31 2021: Interface veth57e672a4 added Fri Sep 17 19:12:36 2021: Interface vethb449227a added Fri Sep 17 19:12:50 2021: Interface veth79eabe8e added Fri Sep 17 19:13:03 2021: Interface vethba75db4e added Fri Sep 17 19:13:14 2021: Interface veth7135e58f deleted Fri Sep 17 19:13:14 2021: Interface veth22b381d1 deleted Fri Sep 17 19:13:14 2021: Interface veth9cdaded6 deleted Fri Sep 17 19:13:15 2021: Interface vethdb837fe2 deleted Fri Sep 17 19:13:15 2021: Interface veth4926b2b3 deleted Fri Sep 17 19:13:15 2021: Interface vetha9ba698c deleted Fri Sep 17 19:13:15 2021: Interface vethc765b21f deleted Fri Sep 17 19:14:02 2021: Interface veth90280fef added Fri Sep 17 19:14:04 2021: Interface vethb9abefb7 added Fri Sep 17 19:14:06 2021: Interface veth449f9063 deleted Fri Sep 17 19:14:14 2021: Interface veth285530f8 added Fri Sep 17 19:14:15 2021: Interface vethbccd808b deleted Fri Sep 17 19:14:16 2021: Interface veth57e672a4 deleted Fri Sep 17 19:14:16 2021: Interface veth79eabe8e deleted Fri Sep 17 19:14:16 2021: Interface vethba75db4e deleted Fri Sep 17 19:14:34 2021: Interface veth3440f6e2 added Fri Sep 17 19:14:51 2021: Interface veth9b7f22bf added Fri Sep 17 19:14:52 2021: Interface vethac9b6883 added Fri Sep 17 19:15:20 2021: Interface veth848ecc24 added Fri Sep 17 19:15:37 2021: Interface vetha0255a4e added Fri Sep 17 19:15:37 2021: Interface veth1dab44be added Fri Sep 17 19:15:43 2021: Interface vethab889443 added Fri Sep 17 19:15:58 2021: Interface vethdd03a9e0 added Fri Sep 17 19:16:16 2021: Interface vethd51de0a8 deleted Fri Sep 17 19:16:36 2021: Interface veth6bb4e6d3 added Fri Sep 17 19:16:39 2021: Interface veth2407afca added Fri Sep 17 19:16:51 2021: Interface vethb9abefb7 deleted Fri Sep 17 19:17:16 2021: Interface vethdd03a9e0 deleted Fri Sep 17 19:17:57 2021: (system-testing/ag-1) Receive advertisement timeout Fri Sep 17 19:17:57 2021: (system-testing/ag-1) Entering MASTER STATE Fri Sep 17 19:17:57 2021: (system-testing/ag-1) setting VIPs. Fri Sep 17 19:17:57 2021: (system-testing/ag-1) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.104 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:17:57 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 19:17:57 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 19:17:57 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 19:17:57 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:17:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:17:57 2021: (system-testing/ag-1) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:17:57 2021: (system-testing/ag-1) Entering BACKUP STATE Fri Sep 17 19:17:57 2021: (system-testing/ag-1) removing VIPs. Fri Sep 17 19:17:57 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:17:57 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 19:17:57 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 19:18:12 2021: Interface veth9cbb9092 added Fri Sep 17 19:18:16 2021: Interface veth2407afca deleted Fri Sep 17 19:18:17 2021: Interface veth1dab44be deleted Fri Sep 17 19:18:17 2021: Interface veth6bb4e6d3 deleted Fri Sep 17 19:19:05 2021: Interface veth11d5d909 added Fri Sep 17 19:19:19 2021: Interface vetha9424393 added Fri Sep 17 19:19:19 2021: Interface vethac9b6883 deleted Fri Sep 17 19:19:23 2021: Interface vethc538fc74 added Fri Sep 17 19:19:23 2021: Interface veth90280fef deleted Fri Sep 17 19:19:23 2021: Interface veth285530f8 deleted Fri Sep 17 19:19:23 2021: Interface vethb449227a deleted Fri Sep 17 19:19:23 2021: Interface veth848ecc24 deleted Fri Sep 17 19:19:23 2021: Interface vethf2141fa2 added Fri Sep 17 19:20:27 2021: Interface vetha9424393 deleted Fri Sep 17 19:20:27 2021: Interface veth86b6c908 added Fri Sep 17 19:20:39 2021: (system-testing/ag-1) Receive advertisement timeout Fri Sep 17 19:20:39 2021: (system-testing/ag-1) Entering MASTER STATE Fri Sep 17 19:20:39 2021: (system-testing/ag-1) setting VIPs. Fri Sep 17 19:20:39 2021: (system-testing/ag-1) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.104 Fri Sep 17 19:20:39 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:20:39 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:20:39 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:20:39 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:20:39 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:20:40 2021: (system-testing/ag-1) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:20:40 2021: (system-testing/ag-1) Entering BACKUP STATE Fri Sep 17 19:20:40 2021: (system-testing/ag-1) removing VIPs. Fri Sep 17 19:21:08 2021: Interface veth3456076d added Fri Sep 17 19:21:41 2021: Interface vethfe8e5cba added Fri Sep 17 19:21:49 2021: Interface veth22dda4ab added Fri Sep 17 19:21:54 2021: Interface veth620a3b16 added Fri Sep 17 19:22:23 2021: Interface veth80abb08e added Fri Sep 17 19:22:24 2021: Interface vethbfa35201 added Fri Sep 17 19:22:27 2021: Interface veth57b650cc deleted Fri Sep 17 19:22:27 2021: Interface veth86b6c908 deleted Fri Sep 17 19:22:27 2021: Interface vethf2141fa2 deleted Fri Sep 17 19:22:28 2021: Interface vethc538fc74 deleted Fri Sep 17 19:22:28 2021: Interface veth9cbb9092 deleted Fri Sep 17 19:22:28 2021: Interface veth39091122 deleted Fri Sep 17 19:22:40 2021: Interface veth306d418a added Fri Sep 17 19:23:28 2021: Interface veth80abb08e deleted Fri Sep 17 19:23:28 2021: Interface veth620a3b16 deleted Fri Sep 17 19:24:28 2021: Interface veth3456076d deleted Fri Sep 17 19:24:28 2021: Interface vethfe8e5cba deleted Fri Sep 17 19:25:14 2021: Interface veth24517cfa added Fri Sep 17 19:25:32 2021: Interface veth11d5d909 deleted Fri Sep 17 19:25:32 2021: Interface vethf1f62675 added Fri Sep 17 19:25:32 2021: Interface vethbfa35201 deleted Fri Sep 17 19:25:33 2021: Interface veth22dda4ab deleted Fri Sep 17 19:26:25 2021: Interface vethfbf4a6d6 added Fri Sep 17 19:26:33 2021: Interface veth2b68df22 added Fri Sep 17 19:26:33 2021: Interface vethf1f62675 deleted Fri Sep 17 19:26:43 2021: Interface veth538dec49 added Fri Sep 17 19:26:59 2021: Interface veth094f5c93 added Fri Sep 17 19:27:38 2021: Interface veth9b7f22bf deleted Fri Sep 17 19:27:42 2021: Interface veth877d75e1 added Fri Sep 17 19:27:51 2021: Interface vethb46829ea added Fri Sep 17 19:28:40 2021: Interface veth24517cfa deleted Fri Sep 17 19:28:41 2021: Interface vethab889443 deleted Fri Sep 17 19:28:41 2021: Interface veth877d75e1 deleted Fri Sep 17 19:28:41 2021: Interface veth538dec49 deleted Fri Sep 17 19:28:44 2021: Interface vethd35fb03f added Fri Sep 17 19:29:11 2021: Interface vethcd817e1b added Fri Sep 17 19:29:41 2021: Interface veth29068843 deleted Fri Sep 17 19:29:41 2021: Interface veth306d418a deleted Fri Sep 17 19:29:41 2021: Interface veth094f5c93 deleted Fri Sep 17 19:29:41 2021: Interface veth2b68df22 deleted Fri Sep 17 19:30:10 2021: Interface vethebd19234 added Fri Sep 17 19:30:13 2021: Interface vethfa8e0ba1 added Fri Sep 17 19:30:41 2021: Interface vethb46829ea deleted Fri Sep 17 19:30:52 2021: Interface vethbbb55d04 added Fri Sep 17 19:31:21 2021: Interface veth3f9153b7 added Fri Sep 17 19:31:22 2021: Interface vethd2dbe32a added Fri Sep 17 19:31:41 2021: Interface vethebd19234 deleted Fri Sep 17 19:31:42 2021: Interface vethd35fb03f deleted Fri Sep 17 19:32:02 2021: Interface veth5b108949 added Fri Sep 17 19:32:42 2021: Interface vethfbf4a6d6 deleted Fri Sep 17 19:33:40 2021: Interface veth5e171b51 added Fri Sep 17 19:33:42 2021: Interface vethfa8e0ba1 deleted Fri Sep 17 19:33:42 2021: Interface veth3f9153b7 deleted Fri Sep 17 19:33:53 2021: Interface vethffd81751 added Fri Sep 17 19:34:31 2021: Interface veth97cba97a added Fri Sep 17 19:34:31 2021: Interface vethd7e22463 added Fri Sep 17 19:34:36 2021: Interface veth4d9cdc21 added Fri Sep 17 19:34:48 2021: Interface veth5b108949 deleted Fri Sep 17 19:35:17 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 19:35:17 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 19:35:17 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 19:35:17 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:35:17 2021: (system-testing/ag-1) Receive advertisement timeout Fri Sep 17 19:35:17 2021: (system-testing/ag-1) Entering MASTER STATE Fri Sep 17 19:35:17 2021: (system-testing/ag-1) setting VIPs. Fri Sep 17 19:35:17 2021: (system-testing/ag-1) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.104 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:35:17 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:35:17 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:35:17 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 19:35:17 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 19:35:17 2021: (system-testing/ag-1) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:35:17 2021: (system-testing/ag-1) Entering BACKUP STATE Fri Sep 17 19:35:17 2021: (system-testing/ag-1) removing VIPs. Fri Sep 17 19:35:50 2021: Interface veth5e171b51 deleted Fri Sep 17 19:36:36 2021: Interface vethf61cb309 added Fri Sep 17 19:36:43 2021: Interface veth58699fea added Fri Sep 17 19:36:51 2021: Interface vethffd81751 deleted Fri Sep 17 19:37:15 2021: Interface veth16bb12fe added Fri Sep 17 19:37:20 2021: Interface veth84a7c504 added Fri Sep 17 19:37:50 2021: Interface veth7b3b603d added Fri Sep 17 19:37:50 2021: Interface vethed5edfde added Fri Sep 17 19:37:50 2021: Interface veth6e516546 added Fri Sep 17 19:37:51 2021: Interface vethd7e22463 deleted Fri Sep 17 19:37:51 2021: Interface vethbbb55d04 deleted Fri Sep 17 19:37:51 2021: Interface vethcd817e1b deleted Fri Sep 17 19:37:51 2021: Interface vethd2dbe32a deleted Fri Sep 17 19:37:52 2021: Interface veth4d9cdc21 deleted Fri Sep 17 19:38:09 2021: Interface veth67ab5f64 added Fri Sep 17 19:38:20 2021: Interface veth808c86fa added Fri Sep 17 19:38:21 2021: Interface veth20ddcee5 added Fri Sep 17 19:38:29 2021: Interface veth9e27488c added Fri Sep 17 19:38:52 2021: Interface veth6e516546 deleted Fri Sep 17 19:38:53 2021: Interface veth97cba97a deleted Fri Sep 17 19:39:07 2021: Reloading ... Fri Sep 17 19:39:07 2021: Opening file '/etc/keepalived.d/keepalived.conf'. Fri Sep 17 19:39:07 2021: Reloading Fri Sep 17 19:39:07 2021: Opening file '/etc/keepalived.d/keepalived.conf'. Fri Sep 17 19:39:07 2021: Assigned address 10.22.5.29 for interface ens192 Fri Sep 17 19:39:07 2021: Assigned address fe80::a20b:9ca2:9be:8470 for interface ens192 Fri Sep 17 19:39:07 2021: (at-sys/backoffice-ip-elm) Entering BACKUP STATE (init) Fri Sep 17 19:39:07 2021: VRRP sockpool: [ifindex( 2), family(IPv4), proto(112), fd(10,11)] Fri Sep 17 19:39:09 2021: Interface vethcad7e8a0 added Fri Sep 17 19:39:53 2021: Interface vethf61cb309 deleted Fri Sep 17 19:39:56 2021: Interface veth9e27488c deleted Fri Sep 17 19:40:00 2021: Interface veth26d54a29 added Fri Sep 17 19:40:03 2021: Interface vetha35e273f added Fri Sep 17 19:40:04 2021: Interface vethc37111c3 added Fri Sep 17 19:40:33 2021: Interface vethb2cec12e added Fri Sep 17 19:40:38 2021: Reloading ... Fri Sep 17 19:40:38 2021: Opening file '/etc/keepalived.d/keepalived.conf'. Fri Sep 17 19:40:38 2021: Reloading Fri Sep 17 19:40:38 2021: Opening file '/etc/keepalived.d/keepalived.conf'. Fri Sep 17 19:40:38 2021: Assigned address 10.22.5.29 for interface ens192 Fri Sep 17 19:40:38 2021: Assigned address fe80::a20b:9ca2:9be:8470 for interface ens192 Fri Sep 17 19:40:38 2021: (at-sys/app-classd-server-ext-001-0) Entering BACKUP STATE (init) Fri Sep 17 19:40:38 2021: VRRP sockpool: [ifindex( 2), family(IPv4), proto(112), fd(10,11)] Fri Sep 17 19:40:42 2021: (at-sys/app-classd-server-ext-001-0) Receive advertisement timeout Fri Sep 17 19:40:42 2021: (at-sys/app-classd-server-ext-001-0) Entering MASTER STATE Fri Sep 17 19:40:42 2021: (at-sys/app-classd-server-ext-001-0) setting VIPs. Fri Sep 17 19:40:42 2021: (at-sys/app-classd-server-ext-001-0) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.119 Fri Sep 17 19:40:42 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:42 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:42 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:42 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:42 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:42 2021: Interface vethac0383ee added Fri Sep 17 19:40:47 2021: (at-sys/app-classd-server-ext-001-0) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.119 Fri Sep 17 19:40:47 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:47 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:47 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:47 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:47 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:40:51 2021: Interface vethb80e275e added Fri Sep 17 19:40:54 2021: Interface vethed5edfde deleted Fri Sep 17 19:40:54 2021: Interface veth16bb12fe deleted Fri Sep 17 19:40:54 2021: Interface veth808c86fa deleted Fri Sep 17 19:40:54 2021: Interface veth67ab5f64 deleted Fri Sep 17 19:40:54 2021: Interface veth7b3b603d deleted Fri Sep 17 19:40:54 2021: Interface veth20ddcee5 deleted Fri Sep 17 19:40:54 2021: Interface veth84a7c504 deleted Fri Sep 17 19:40:58 2021: (at-sys/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 19:40:58 2021: (at-sys/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 19:40:58 2021: (at-sys/backoffice-ip-elm) setting VIPs. Fri Sep 17 19:40:58 2021: (at-sys/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.103 Fri Sep 17 19:40:58 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:40:58 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:40:58 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:40:58 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:40:58 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:40:58 2021: (at-sys/backoffice-ip-elm) Master received advert from 10.22.5.38 with same priority 100 but higher IP address than ours Fri Sep 17 19:40:58 2021: (at-sys/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 19:40:58 2021: (at-sys/backoffice-ip-elm) removing VIPs. Fri Sep 17 19:41:16 2021: Interface veth18e6ae50 added Fri Sep 17 19:41:23 2021: (at-sys/app-classd-server-ext-001-0) Received advert from 10.22.5.23 with lower priority 100, ours 100, forcing new election Fri Sep 17 19:41:23 2021: (at-sys/app-classd-server-ext-001-0) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.119 Fri Sep 17 19:41:23 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:23 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:23 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:23 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:23 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:28 2021: (at-sys/app-classd-server-ext-001-0) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.119 Fri Sep 17 19:41:28 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:28 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:28 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:28 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:28 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:41:32 2021: Interface veth70ccc24e added Fri Sep 17 19:41:37 2021: Interface vethfaa83982 added Fri Sep 17 19:41:46 2021: (at-sys/app-classd-server-ext-001-0) Master received advert from 10.22.5.31 with same priority 100 but higher IP address than ours Fri Sep 17 19:41:46 2021: (at-sys/app-classd-server-ext-001-0) Entering BACKUP STATE Fri Sep 17 19:41:46 2021: (at-sys/app-classd-server-ext-001-0) removing VIPs. Fri Sep 17 19:41:49 2021: Interface vethcb368fe1 added Fri Sep 17 19:41:54 2021: Interface vethcad7e8a0 deleted Fri Sep 17 19:41:54 2021: Interface vethc37111c3 deleted Fri Sep 17 19:41:55 2021: Interface vetha0255a4e deleted Fri Sep 17 19:42:06 2021: Interface veth22cedb1b added Fri Sep 17 19:42:10 2021: Interface veth3b414201 added Fri Sep 17 19:42:49 2021: Interface veth5999099a added Fri Sep 17 19:43:06 2021: Interface veth934aab32 added Fri Sep 17 19:43:07 2021: Interface veth9de324a9 added Fri Sep 17 19:43:09 2021: Interface veth1eadc288 added Fri Sep 17 19:43:45 2021: Interface veth9416f4cd added Fri Sep 17 19:43:49 2021: Interface vethdb506ee8 added Fri Sep 17 19:43:56 2021: Interface vetha35e273f deleted Fri Sep 17 19:43:56 2021: Interface veth22cedb1b deleted Fri Sep 17 19:43:56 2021: Interface veth26d54a29 deleted Fri Sep 17 19:44:10 2021: Interface veth83386d1e added Fri Sep 17 19:44:14 2021: Interface veth8ec64378 added Fri Sep 17 19:44:40 2021: Interface veth79789958 added Fri Sep 17 19:44:49 2021: Interface veth574c2127 added Fri Sep 17 19:44:49 2021: Interface veth354c3666 added Fri Sep 17 19:44:53 2021: Interface vethe48ef806 added Fri Sep 17 19:44:53 2021: Interface vetha68fe603 added Fri Sep 17 19:44:56 2021: Interface vethe251c7b9 added Fri Sep 17 19:44:56 2021: Interface vethcb368fe1 deleted Fri Sep 17 19:44:56 2021: Interface vethfaa83982 deleted Fri Sep 17 19:44:57 2021: Interface vethb80e275e deleted Fri Sep 17 19:44:57 2021: Interface veth5999099a deleted Fri Sep 17 19:44:57 2021: Interface vethac0383ee deleted Fri Sep 17 19:44:57 2021: Interface veth18e6ae50 deleted Fri Sep 17 19:44:59 2021: Interface veth65d8b68c added Fri Sep 17 19:45:06 2021: Interface vetha0d7fd12 added Fri Sep 17 19:45:15 2021: Interface veth4ac59d12 added Fri Sep 17 19:45:16 2021: Interface veth23389745 added Fri Sep 17 19:45:40 2021: Interface vetha1937fda added Fri Sep 17 19:45:53 2021: Interface vethc58482b7 added Fri Sep 17 19:45:55 2021: Interface veth1eadc288 deleted Fri Sep 17 19:45:55 2021: Interface vethf70ce34d added Fri Sep 17 19:45:57 2021: Interface vethdb506ee8 deleted Fri Sep 17 19:45:57 2021: Interface veth70ccc24e deleted Fri Sep 17 19:45:57 2021: Interface veth3b414201 deleted Fri Sep 17 19:46:01 2021: Interface veth5d702a90 added Fri Sep 17 19:46:01 2021: Interface veth9416f4cd deleted Fri Sep 17 19:46:04 2021: Interface vetha443bd6d added Fri Sep 17 19:46:06 2021: Interface veth575f3681 added Fri Sep 17 19:46:50 2021: Interface veth9de324a9 deleted Fri Sep 17 19:46:58 2021: Interface vethe48ef806 deleted Fri Sep 17 19:46:58 2021: Interface vethb2cec12e deleted Fri Sep 17 19:46:58 2021: Interface veth8ec64378 deleted Fri Sep 17 19:47:01 2021: Interface veth8c96b0f6 added Fri Sep 17 19:47:20 2021: Interface veth0a397ecb added Fri Sep 17 19:47:29 2021: Interface vethf133df46 added Fri Sep 17 19:47:33 2021: Interface veth6b20ca3f added Fri Sep 17 19:47:42 2021: Interface veth934aab32 deleted Fri Sep 17 19:47:53 2021: Interface vethb74ac673 added Fri Sep 17 19:47:58 2021: Interface veth83386d1e deleted Fri Sep 17 19:47:58 2021: Interface veth79789958 deleted Fri Sep 17 19:48:14 2021: Interface vethe9396da8 added Fri Sep 17 19:48:18 2021: Interface vethd02d224c added Fri Sep 17 19:48:22 2021: Interface veth63fd4c4f added Fri Sep 17 19:48:22 2021: Interface veth039cbdf0 added Fri Sep 17 19:48:22 2021: Interface vethb04e36f5 added Fri Sep 17 19:48:22 2021: Interface veth00c12df2 added Fri Sep 17 19:48:22 2021: Interface vethae7261f0 added Fri Sep 17 19:48:50 2021: Interface veth742e5e05 added Fri Sep 17 19:48:50 2021: Interface veth279b8da0 added Fri Sep 17 19:48:54 2021: Interface vethbdf54abd added Fri Sep 17 19:48:58 2021: Interface veth575f3681 deleted Fri Sep 17 19:48:59 2021: Interface veth65d8b68c deleted Fri Sep 17 19:48:59 2021: Interface vethf133df46 deleted Fri Sep 17 19:48:59 2021: Interface veth23389745 deleted Fri Sep 17 19:48:59 2021: Interface veth354c3666 deleted Fri Sep 17 19:48:59 2021: Interface veth5d702a90 deleted Fri Sep 17 19:48:59 2021: Interface vetha0d7fd12 deleted Fri Sep 17 19:48:59 2021: Interface veth8c96b0f6 deleted Fri Sep 17 19:49:02 2021: Interface vetha1937fda deleted Fri Sep 17 19:49:39 2021: Interface veth4acdffd1 added Fri Sep 17 19:49:59 2021: Interface veth574c2127 deleted Fri Sep 17 19:50:00 2021: Interface veth279b8da0 deleted Fri Sep 17 19:50:00 2021: Interface vetha443bd6d deleted Fri Sep 17 19:50:00 2021: Interface vethf70ce34d deleted Fri Sep 17 19:50:00 2021: Interface veth4ac59d12 deleted Fri Sep 17 19:50:00 2021: Interface veth6b20ca3f deleted Fri Sep 17 19:50:28 2021: Interface veth08c955fe added Fri Sep 17 19:50:33 2021: Interface veth5e8b63c6 added Fri Sep 17 19:50:56 2021: Interface veth9c5d7a14 added Fri Sep 17 19:50:57 2021: Interface veth38f9cf99 added Fri Sep 17 19:51:00 2021: Interface veth0a397ecb deleted Fri Sep 17 19:51:01 2021: Interface vethd02d224c deleted Fri Sep 17 19:51:01 2021: Interface vetha68fe603 deleted Fri Sep 17 19:51:01 2021: Interface vethe251c7b9 deleted Fri Sep 17 19:51:01 2021: Interface vethe9396da8 deleted Fri Sep 17 19:52:01 2021: Interface veth00c12df2 deleted Fri Sep 17 19:52:01 2021: Interface vethc58482b7 deleted Fri Sep 17 19:52:01 2021: Interface veth3440f6e2 deleted Fri Sep 17 19:52:02 2021: Interface veth039cbdf0 deleted Fri Sep 17 19:52:02 2021: Interface vethc6805096 added Fri Sep 17 19:52:02 2021: Interface veth12e67b91 added Fri Sep 17 19:52:02 2021: Interface veth63fd4c4f deleted Fri Sep 17 19:52:04 2021: Interface veth63cf7d9d added Fri Sep 17 19:52:29 2021: Interface veth4eafaa0f added Fri Sep 17 19:52:29 2021: Interface veth8dd24eda added Fri Sep 17 19:52:34 2021: Interface vethe36fba39 added Fri Sep 17 19:53:05 2021: Interface veth87ad9127 added Fri Sep 17 19:53:05 2021: Interface vethcb83a10d added Fri Sep 17 19:53:05 2021: Interface veth08c955fe deleted Fri Sep 17 19:53:05 2021: Interface veth185bea58 added Fri Sep 17 19:53:12 2021: Interface veth9e9913fa added Fri Sep 17 19:53:21 2021: Interface veth0c680b64 added Fri Sep 17 19:53:49 2021: Interface veth9bd5fdb9 added Fri Sep 17 19:53:49 2021: Interface veth15127998 added Fri Sep 17 19:53:51 2021: Interface veth1c752893 added Fri Sep 17 19:53:56 2021: Interface veth1b76b7c9 added Fri Sep 17 19:54:04 2021: Interface veth6c19026d added Fri Sep 17 19:54:04 2021: Interface veth87ad9127 deleted Fri Sep 17 19:54:05 2021: Interface veth9c5d7a14 deleted Fri Sep 17 19:54:05 2021: Interface vethf1eec697 added Fri Sep 17 19:54:05 2021: Interface vethc6805096 deleted Fri Sep 17 19:54:05 2021: Interface veth622db8d3 added Fri Sep 17 19:54:05 2021: Interface veth63cf7d9d deleted Fri Sep 17 19:54:05 2021: Interface vethb74ac673 deleted Fri Sep 17 19:54:05 2021: Interface vethb04e36f5 deleted Fri Sep 17 19:54:05 2021: Interface veth4eafaa0f deleted Fri Sep 17 19:54:31 2021: Interface veth53fa7ceb added Fri Sep 17 19:54:31 2021: Interface veth75faf50e added Fri Sep 17 19:54:31 2021: Interface vetha2664c99 added Fri Sep 17 19:55:06 2021: Interface veth742e5e05 deleted Fri Sep 17 19:55:06 2021: Interface veth185bea58 deleted Fri Sep 17 19:55:06 2021: Interface vethae7261f0 deleted Fri Sep 17 19:55:06 2021: Interface veth38f9cf99 deleted Fri Sep 17 19:55:06 2021: Interface vethbdf54abd deleted Fri Sep 17 19:55:06 2021: Interface veth5e8b63c6 deleted Fri Sep 17 19:55:20 2021: Interface veth8dc3c3e7 added Fri Sep 17 19:55:30 2021: Interface vethfe016305 added Fri Sep 17 19:55:34 2021: Interface vethe5d59723 added Fri Sep 17 19:55:36 2021: Interface veth828d9e11 added Fri Sep 17 19:55:39 2021: Interface vethd0358ad6 added Fri Sep 17 19:55:39 2021: Interface veth06047b2d added Fri Sep 17 19:55:59 2021: Interface veth444decf3 added Fri Sep 17 19:55:59 2021: Interface vethb606e8f8 added Fri Sep 17 19:56:06 2021: Interface veth4acdffd1 deleted Fri Sep 17 19:56:07 2021: Interface veth12e67b91 deleted Fri Sep 17 19:56:07 2021: Interface veth622db8d3 deleted Fri Sep 17 19:56:07 2021: Interface veth9e9913fa deleted Fri Sep 17 19:56:07 2021: Interface vethe36fba39 deleted Fri Sep 17 19:56:07 2021: Interface veth75faf50e deleted Fri Sep 17 19:56:07 2021: Interface veth8dc3c3e7 deleted Fri Sep 17 19:56:08 2021: Interface veth1543318b added Fri Sep 17 19:56:08 2021: Interface veth8af72309 added Fri Sep 17 19:56:17 2021: Interface veth267eb45c added Fri Sep 17 19:56:29 2021: Interface vetha31befac added Fri Sep 17 19:56:30 2021: Interface vethfd7876ff added Fri Sep 17 19:56:59 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 19:56:59 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 19:56:59 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 19:56:59 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:56:59 2021: (system-testing/ag-1) Receive advertisement timeout Fri Sep 17 19:56:59 2021: (system-testing/ag-1) Entering MASTER STATE Fri Sep 17 19:56:59 2021: (system-testing/ag-1) setting VIPs. Fri Sep 17 19:56:59 2021: (system-testing/ag-1) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.104 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 19:56:59 2021: (at-sys/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 19:56:59 2021: (at-sys/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 19:56:59 2021: (at-sys/backoffice-ip-elm) setting VIPs. Fri Sep 17 19:56:59 2021: (at-sys/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.103 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 19:56:59 2021: (at-sys/app-classd-server-ext-001-0) Receive advertisement timeout Fri Sep 17 19:56:59 2021: (at-sys/app-classd-server-ext-001-0) Entering MASTER STATE Fri Sep 17 19:56:59 2021: (at-sys/app-classd-server-ext-001-0) setting VIPs. Fri Sep 17 19:56:59 2021: (at-sys/app-classd-server-ext-001-0) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.119 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:56:59 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 19:56:59 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:56:59 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 19:56:59 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 19:56:59 2021: (system-testing/ag-1) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:56:59 2021: (system-testing/ag-1) Entering BACKUP STATE Fri Sep 17 19:56:59 2021: (system-testing/ag-1) removing VIPs. Fri Sep 17 19:56:59 2021: (at-sys/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:56:59 2021: (at-sys/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 19:56:59 2021: (at-sys/backoffice-ip-elm) removing VIPs. Fri Sep 17 19:56:59 2021: (at-sys/app-classd-server-ext-001-0) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:56:59 2021: (at-sys/app-classd-server-ext-001-0) Entering BACKUP STATE Fri Sep 17 19:56:59 2021: (at-sys/app-classd-server-ext-001-0) removing VIPs. Fri Sep 17 19:57:21 2021: Interface vethfe016305 deleted Fri Sep 17 19:57:21 2021: Interface veth09b35663 added Fri Sep 17 19:57:21 2021: Interface veth18189ba2 added Fri Sep 17 19:57:21 2021: Interface vethe8c0a0d4 added Fri Sep 17 19:57:21 2021: Interface veth3adb8b96 added Fri Sep 17 19:57:21 2021: Interface veth48e0908e added Fri Sep 17 19:57:28 2021: Interface vethf310dbd9 added Fri Sep 17 19:57:28 2021: Interface veth58df965b added Fri Sep 17 19:57:55 2021: Interface vethced5f008 added Fri Sep 17 19:57:55 2021: Interface veth76d79390 added Fri Sep 17 19:57:55 2021: Interface vethd20122b6 added Fri Sep 17 19:58:19 2021: Interface veth444decf3 deleted Fri Sep 17 19:58:19 2021: Interface vethf7cebb2b added Fri Sep 17 19:58:20 2021: Interface veth3adb8b96 deleted Fri Sep 17 19:58:20 2021: Interface veth58d91340 added Fri Sep 17 19:58:20 2021: Interface vethd8e8518b added Fri Sep 17 19:58:20 2021: Interface veth9bd5fdb9 deleted Fri Sep 17 19:58:20 2021: Interface veth1b76b7c9 deleted Fri Sep 17 19:58:20 2021: Interface vetha2664c99 deleted Fri Sep 17 19:58:20 2021: Interface veth1c752893 deleted Fri Sep 17 19:58:21 2021: Interface veth6c19026d deleted Fri Sep 17 19:58:21 2021: Interface veth53fa7ceb deleted Fri Sep 17 19:58:21 2021: Interface veth267eb45c deleted Fri Sep 17 19:58:21 2021: Interface veth06047b2d deleted Fri Sep 17 19:58:32 2021: Interface veth77e24fd8 added Fri Sep 17 19:58:35 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 19:58:35 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 19:58:35 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 19:58:36 2021: Interface veth6d9b37c4 added Fri Sep 17 19:58:36 2021: Interface veth827ab25c added Fri Sep 17 19:58:36 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 19:58:36 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:58:36 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:58:36 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:58:36 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:58:36 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 19:58:36 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 19:58:36 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 19:58:36 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 19:58:36 2021: Interface vetha6d3dfd9 added Fri Sep 17 19:58:36 2021: Interface vethff2c4124 added Fri Sep 17 19:58:36 2021: Interface vethf73efd21 added Fri Sep 17 19:58:37 2021: Interface veth2d6fc2c0 added Fri Sep 17 19:59:07 2021: Interface veth7426981c added Fri Sep 17 19:59:07 2021: Interface veth895323ca added Fri Sep 17 19:59:07 2021: Interface veth3436d1c6 added Fri Sep 17 19:59:07 2021: Interface vetha2a21389 added Fri Sep 17 19:59:14 2021: Interface veth46668077 added Fri Sep 17 19:59:21 2021: Interface vethb606e8f8 deleted Fri Sep 17 19:59:21 2021: Interface vethd0358ad6 deleted Fri Sep 17 19:59:22 2021: Interface vethe5d59723 deleted Fri Sep 17 19:59:22 2021: Interface veth8af72309 deleted Fri Sep 17 19:59:22 2021: Interface vethcb83a10d deleted Fri Sep 17 19:59:22 2021: Interface vethfd7876ff deleted Fri Sep 17 19:59:22 2021: Interface veth8dd24eda deleted Fri Sep 17 19:59:22 2021: Interface vethced5f008 deleted Fri Sep 17 20:00:19 2021: Interface veth0c680b64 deleted Fri Sep 17 20:00:22 2021: Interface veth827ab25c deleted Fri Sep 17 20:00:23 2021: Interface vethf310dbd9 deleted Fri Sep 17 20:00:23 2021: Interface veth15127998 deleted Fri Sep 17 20:00:35 2021: Interface vethd20122b6 deleted Fri Sep 17 20:00:47 2021: Interface vethc785af1b added Fri Sep 17 20:00:50 2021: Interface veth18189ba2 deleted Fri Sep 17 20:01:12 2021: Interface vetha31befac deleted Fri Sep 17 20:01:23 2021: Interface vethf1eec697 deleted Fri Sep 17 20:01:23 2021: Interface veth58df965b deleted Fri Sep 17 20:01:23 2021: Interface veth09b35663 deleted Fri Sep 17 20:01:23 2021: Interface vethe8c0a0d4 deleted Fri Sep 17 20:01:24 2021: Interface vethd8e8518b deleted Fri Sep 17 20:01:24 2021: Interface vetha2a21389 deleted Fri Sep 17 20:02:25 2021: Interface vethf84871af added Fri Sep 17 20:02:25 2021: Interface veth76d79390 deleted Fri Sep 17 20:02:27 2021: Interface vethf73efd21 deleted Fri Sep 17 20:02:27 2021: Interface vethf7cebb2b deleted Fri Sep 17 20:02:27 2021: Interface veth1543318b deleted Fri Sep 17 20:02:27 2021: Interface veth46668077 deleted Fri Sep 17 20:02:27 2021: Interface veth828d9e11 deleted Fri Sep 17 20:02:41 2021: Interface veth79014206 added Fri Sep 17 20:02:44 2021: Interface veth134a7702 added Fri Sep 17 20:02:44 2021: Interface vethe71abb25 added Fri Sep 17 20:02:55 2021: Interface veth57dcf97e added Fri Sep 17 20:03:00 2021: Interface veth93a8965a added Fri Sep 17 20:03:04 2021: Interface veth765e0f11 added Fri Sep 17 20:03:22 2021: Interface veth1945896c added Fri Sep 17 20:03:29 2021: Interface veth79014206 deleted Fri Sep 17 20:03:29 2021: Interface veth6d9b37c4 deleted Fri Sep 17 20:03:29 2021: Interface veth7426981c deleted Fri Sep 17 20:03:29 2021: Interface vethc785af1b deleted Fri Sep 17 20:03:30 2021: Interface veth895323ca deleted Fri Sep 17 20:03:30 2021: Interface veth77e24fd8 deleted Fri Sep 17 20:03:32 2021: Interface veth887f5ea8 added Fri Sep 17 20:03:33 2021: Interface veth32298d6c added Fri Sep 17 20:03:34 2021: Interface veth5c040263 added Fri Sep 17 20:03:47 2021: Interface vethd249c100 added Fri Sep 17 20:04:03 2021: Interface veth864cd444 added Fri Sep 17 20:04:07 2021: Interface veth53508aa7 added Fri Sep 17 20:04:18 2021: Interface veth3c1361f8 added Fri Sep 17 20:04:29 2021: Interface veth69be9092 added Fri Sep 17 20:04:29 2021: Interface vetha27e8460 added Fri Sep 17 20:04:32 2021: Interface veth3436d1c6 deleted Fri Sep 17 20:04:33 2021: Interface veth1945896c deleted Fri Sep 17 20:04:33 2021: Interface veth2d6fc2c0 deleted Fri Sep 17 20:04:33 2021: Interface veth48e0908e deleted Fri Sep 17 20:04:40 2021: Interface veth9c6022af added Fri Sep 17 20:04:41 2021: Interface vetha4c7d3d4 added Fri Sep 17 20:05:05 2021: Interface veth554186b2 added Fri Sep 17 20:05:09 2021: Interface veth227351bc added Fri Sep 17 20:05:28 2021: Interface vethcae815da added Fri Sep 17 20:05:29 2021: Interface veth67a6809d added Fri Sep 17 20:05:33 2021: Interface vetha6d3dfd9 deleted Fri Sep 17 20:05:36 2021: Interface veth93a8965a deleted Fri Sep 17 20:05:40 2021: Interface veth765e0f11 deleted Fri Sep 17 20:05:40 2021: Interface vethff2c4124 deleted Fri Sep 17 20:05:40 2021: Interface veth3c1361f8 deleted Fri Sep 17 20:06:14 2021: Interface veth66f97ed1 added Fri Sep 17 20:06:14 2021: Interface vethe3eba138 added Fri Sep 17 20:06:20 2021: Interface vethec299606 added Fri Sep 17 20:06:30 2021: Interface veth7f69bb16 added Fri Sep 17 20:06:30 2021: Interface vethdc7967a1 added Fri Sep 17 20:06:41 2021: Interface veth134a7702 deleted Fri Sep 17 20:06:45 2021: Interface vethe71abb25 deleted Fri Sep 17 20:06:46 2021: Interface veth227351bc deleted Fri Sep 17 20:06:46 2021: Interface veth9c6022af deleted Fri Sep 17 20:06:46 2021: Interface veth32298d6c deleted Fri Sep 17 20:06:46 2021: Interface veth554186b2 deleted Fri Sep 17 20:06:46 2021: Interface veth57dcf97e deleted Fri Sep 17 20:06:49 2021: Interface veth2c6e57cb added Fri Sep 17 20:06:50 2021: Interface vethd25b39ba added Fri Sep 17 20:07:47 2021: Interface vetha4c7d3d4 deleted Fri Sep 17 20:07:50 2021: Interface vethd249c100 deleted Fri Sep 17 20:07:51 2021: Interface veth66f97ed1 deleted Fri Sep 17 20:07:51 2021: Interface veth58d91340 deleted Fri Sep 17 20:07:51 2021: Interface veth864cd444 deleted Fri Sep 17 20:07:51 2021: Interface vethcae815da deleted Fri Sep 17 20:07:51 2021: Interface vetha27e8460 deleted Fri Sep 17 20:08:38 2021: Interface veth11708564 added Fri Sep 17 20:08:42 2021: Interface veth5ae2fc4c added Fri Sep 17 20:08:51 2021: Interface veth53508aa7 deleted Fri Sep 17 20:08:51 2021: Interface vethe3eba138 deleted Fri Sep 17 20:08:51 2021: Interface veth69be9092 deleted Fri Sep 17 20:08:52 2021: Interface vethf84871af deleted Fri Sep 17 20:08:52 2021: Interface veth5c040263 deleted Fri Sep 17 20:09:25 2021: Interface vethd2a0ac0a added Fri Sep 17 20:09:25 2021: Interface veth4b18f77e added Fri Sep 17 20:09:27 2021: Interface vethb49dc6df added Fri Sep 17 20:09:43 2021: Interface veth3da91cfd added Fri Sep 17 20:09:43 2021: Interface veth1a6ba9db added Fri Sep 17 20:09:53 2021: Interface veth2c6e57cb deleted Fri Sep 17 20:09:53 2021: Interface veth887f5ea8 deleted Fri Sep 17 20:09:55 2021: Interface vethc252c3c4 added Fri Sep 17 20:10:00 2021: Interface veth78c36bc5 added Fri Sep 17 20:10:00 2021: Interface vethfff525aa added Fri Sep 17 20:10:04 2021: Interface veth30c6471b added Fri Sep 17 20:10:41 2021: Interface veth33947841 added Fri Sep 17 20:10:50 2021: Interface veth19768d8e added Fri Sep 17 20:10:50 2021: Interface vethee034598 added Fri Sep 17 20:10:51 2021: Interface veth86199167 added Fri Sep 17 20:10:53 2021: Interface veth7f69bb16 deleted Fri Sep 17 20:10:53 2021: Interface vethec299606 deleted Fri Sep 17 20:10:53 2021: Interface veth3da91cfd deleted Fri Sep 17 20:10:53 2021: Interface vethdc7967a1 deleted Fri Sep 17 20:10:54 2021: Interface veth1a6ba9db deleted Fri Sep 17 20:11:00 2021: Interface vethae1a66af added Fri Sep 17 20:11:22 2021: Interface vetha68efe7b added Fri Sep 17 20:11:34 2021: Interface veth59df4f33 added Fri Sep 17 20:11:49 2021: Interface vetha66f485f added Fri Sep 17 20:11:50 2021: Interface veth558c7160 added Fri Sep 17 20:11:52 2021: Interface veth8df8b2e8 added Fri Sep 17 20:11:54 2021: Interface vethd2a0ac0a deleted Fri Sep 17 20:11:54 2021: Interface veth4b18f77e deleted Fri Sep 17 20:11:54 2021: Interface veth19768d8e deleted Fri Sep 17 20:11:54 2021: Interface vethc252c3c4 deleted Fri Sep 17 20:11:54 2021: Interface veth78c36bc5 deleted Fri Sep 17 20:11:57 2021: Interface vethb49dc6df deleted Fri Sep 17 20:12:01 2021: Interface vethe06063e9 added Fri Sep 17 20:12:02 2021: Interface vethde739339 added Fri Sep 17 20:12:10 2021: Interface vethadd530df added Fri Sep 17 20:12:54 2021: Interface veth6b45f8d8 added Fri Sep 17 20:12:54 2021: Interface veth5f04f47f added Fri Sep 17 20:12:55 2021: Interface veth3f05776b added Fri Sep 17 20:12:57 2021: Interface veth2814079e added Fri Sep 17 20:12:58 2021: Interface vetha66f485f deleted Fri Sep 17 20:12:58 2021: Interface vethee034598 deleted Fri Sep 17 20:12:58 2021: Interface vethd25b39ba deleted Fri Sep 17 20:12:58 2021: Interface vetha68efe7b deleted Fri Sep 17 20:12:58 2021: Interface veth30c6471b deleted Fri Sep 17 20:13:09 2021: Interface veth0daadc9c added Fri Sep 17 20:13:17 2021: Interface veth67a6809d deleted Fri Sep 17 20:13:20 2021: (system-testing/ag-1) Receive advertisement timeout Fri Sep 17 20:13:20 2021: (system-testing/ag-1) Entering MASTER STATE Fri Sep 17 20:13:20 2021: (system-testing/ag-1) setting VIPs. Fri Sep 17 20:13:20 2021: (system-testing/ag-1) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.104 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:13:20 2021: (system-testing/ag-1) Master received advert from 10.22.5.38 with same priority 100 but higher IP address than ours Fri Sep 17 20:13:20 2021: (system-testing/ag-1) Entering BACKUP STATE Fri Sep 17 20:13:20 2021: (system-testing/ag-1) removing VIPs. Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Received advert from 10.22.5.28 with lower priority 100, ours 100, forcing new election Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Received advert from 10.22.5.24 with lower priority 100, ours 100, forcing new election Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.38 with same priority 100 but higher IP address than ours Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 20:13:20 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 20:13:22 2021: Interface veth33947841 deleted Fri Sep 17 20:13:25 2021: Interface veth94ba4094 added Fri Sep 17 20:13:56 2021: Interface veth6ae44e7e added Fri Sep 17 20:13:58 2021: Interface veth558c7160 deleted Fri Sep 17 20:13:58 2021: Interface vethe06063e9 deleted Fri Sep 17 20:13:58 2021: Interface veth0daadc9c deleted Fri Sep 17 20:14:15 2021: Interface veth9c1e7ae6 added Fri Sep 17 20:14:24 2021: Interface veth558d418c added Fri Sep 17 20:14:52 2021: Interface veth94442350 added Fri Sep 17 20:14:55 2021: Interface vethc3c43c63 added Fri Sep 17 20:14:59 2021: Interface veth6b45f8d8 deleted Fri Sep 17 20:14:59 2021: Interface vethadd530df deleted Fri Sep 17 20:14:59 2021: Interface veth2814079e deleted Fri Sep 17 20:14:59 2021: Interface veth86199167 deleted Fri Sep 17 20:14:59 2021: Interface veth11708564 deleted Fri Sep 17 20:14:59 2021: Interface veth5ae2fc4c deleted Fri Sep 17 20:15:09 2021: Interface veth042a3533 added Fri Sep 17 20:15:24 2021: Interface veth6f288e90 added Fri Sep 17 20:15:33 2021: Interface veth75e3b813 added Fri Sep 17 20:15:59 2021: Interface veth9eb6491a added Fri Sep 17 20:15:59 2021: Interface veth9c1e7ae6 deleted Fri Sep 17 20:16:00 2021: Interface veth558d418c deleted Fri Sep 17 20:16:00 2021: Interface veth6ae44e7e deleted Fri Sep 17 20:16:07 2021: (at-sys/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 20:16:07 2021: (at-sys/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 20:16:07 2021: (at-sys/backoffice-ip-elm) setting VIPs. Fri Sep 17 20:16:07 2021: (at-sys/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.103 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.103 Fri Sep 17 20:16:07 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 20:16:07 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 20:16:07 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 20:16:07 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:16:07 2021: (at-sys/app-classd-server-ext-001-0) Receive advertisement timeout Fri Sep 17 20:16:07 2021: (at-sys/app-classd-server-ext-001-0) Entering MASTER STATE Fri Sep 17 20:16:07 2021: (at-sys/app-classd-server-ext-001-0) setting VIPs. Fri Sep 17 20:16:07 2021: (at-sys/app-classd-server-ext-001-0) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.119 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.119 Fri Sep 17 20:16:07 2021: (system-testing/ag-1) Receive advertisement timeout Fri Sep 17 20:16:07 2021: (system-testing/ag-1) Entering MASTER STATE Fri Sep 17 20:16:07 2021: (system-testing/ag-1) setting VIPs. Fri Sep 17 20:16:07 2021: (system-testing/ag-1) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.104 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:16:07 2021: Sending gratuitous ARP on ens192 for 10.22.4.104 Fri Sep 17 20:16:07 2021: (at-sys/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 20:16:07 2021: (at-sys/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 20:16:07 2021: (at-sys/backoffice-ip-elm) removing VIPs. Fri Sep 17 20:16:07 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 20:16:07 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 20:16:07 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 20:16:07 2021: (at-sys/app-classd-server-ext-001-0) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 20:16:07 2021: (at-sys/app-classd-server-ext-001-0) Entering BACKUP STATE Fri Sep 17 20:16:07 2021: (at-sys/app-classd-server-ext-001-0) removing VIPs. Fri Sep 17 20:16:07 2021: (system-testing/ag-1) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 20:16:07 2021: (system-testing/ag-1) Entering BACKUP STATE Fri Sep 17 20:16:07 2021: (system-testing/ag-1) removing VIPs. Fri Sep 17 20:16:09 2021: Interface veth0f635c19 added Fri Sep 17 20:16:12 2021: Interface vethcede323d added Fri Sep 17 20:17:00 2021: Interface veth5f04f47f deleted Fri Sep 17 20:17:00 2021: Interface veth042a3533 deleted Fri Sep 17 20:17:00 2021: Interface vethfff525aa deleted Fri Sep 17 20:17:14 2021: Interface vethaa779614 added Fri Sep 17 20:17:17 2021: Interface vethcbdae985 added Fri Sep 17 20:17:33 2021: Interface veth4640cdad added Fri Sep 17 20:17:58 2021: Interface veth6209184d added Fri Sep 17 20:18:02 2021: Interface veth6f288e90 deleted Fri Sep 17 20:18:02 2021: Interface veth59df4f33 deleted Fri Sep 17 20:18:03 2021: Interface vethae1a66af deleted Fri Sep 17 20:18:03 2021: Interface veth94ba4094 deleted Fri Sep 17 20:18:15 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 20:18:15 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 20:18:15 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 20:18:16 2021: Interface vethcc8a361b added Fri Sep 17 20:18:16 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 20:18:16 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:18:16 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:18:16 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:18:16 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:18:16 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:18:16 2021: A thread timer expired 1.342858 seconds ago Fri Sep 17 20:18:16 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 20:18:16 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 20:18:16 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs. Fri Sep 17 20:18:32 2021: Interface veth8527e881 added Fri Sep 17 20:18:54 2021: Interface veth9d327263 added Fri Sep 17 20:18:54 2021: Interface veth31edd5eb added Fri Sep 17 20:19:04 2021: Interface vethde739339 deleted Fri Sep 17 20:19:04 2021: Interface vethc3c43c63 deleted Fri Sep 17 20:19:04 2021: Interface veth8df8b2e8 deleted Fri Sep 17 20:19:04 2021: Interface vethcbdae985 deleted Fri Sep 17 20:19:05 2021: Interface veth1e44de76 deleted Fri Sep 17 20:19:05 2021: Interface vethaa779614 deleted Fri Sep 17 20:19:05 2021: Interface veth3f05776b deleted Fri Sep 17 20:19:05 2021: Interface veth9eb6491a deleted Fri Sep 17 20:19:10 2021: Interface veth0ef06abd added Fri Sep 17 20:19:22 2021: Interface vethdc7e58bf added Fri Sep 17 20:19:38 2021: Interface veth28a3b25f added Fri Sep 17 20:19:38 2021: Interface veth039ab7d9 added Fri Sep 17 20:19:44 2021: Interface veth098a4585 added Fri Sep 17 20:19:45 2021: Interface veth8b02efd4 added Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) Receive advertisement timeout Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) Entering MASTER STATE Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) setting VIPs. Fri Sep 17 20:19:57 2021: Interface vethba375d1d added Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) Received advert from 10.22.5.23 with lower priority 100, ours 100, forcing new election Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) Sending/queueing gratuitous ARPs on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: Sending gratuitous ARP on ens192 for 10.22.4.117 Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) Master received advert from 10.22.5.40 with same priority 100 but higher IP address than ours Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) Entering BACKUP STATE Fri Sep 17 20:19:57 2021: (deepa-zrr/backoffice-ip-elm) removing VIPs.

osirisstar-hub commented 2 years ago

Looks like there is a problem in openshift. Network operator is stuck recycling every few seconds. Thinking this is causing some kind of network issue.

osirisstar-hub commented 2 years ago

Went through and drained and rebooted all of the worker nodes one at a time and this resolved the issue.