Test load-balancer scenarios

[acozine]

• [x] If one load balancer is down, and we restart the other load balancer, what happens?

[carolyncole]

@kayiwa and I ran and experiment this morning.

1. Turn off nginx on lib-adc2 sudo systemctl stop nginx
2. Restart Nginx on lib-adc1 ten times sudo systemctl restart nginx
   1. Check journalctl for errors on lib-adc1 sudo journalctl -u nginx -f
   2. Check ip a for 146 on lib-adc1 ip a
3. Turn on nginx on lib-adc2 sudo systemctl start nginx
4. Restart nginx on lib-adc1 three times sudo systemctl restart nginx
   1. Check journalctl for errors
   2. Check ip a for 146 on lib-adc1

The intention was to try the restart of lib-adc1 until it swapped to make sure that ten restarts was enough to trigger keepalived. We only had to restart 3 times before it swapped. We realized after the fact that we could check journalctl for keepalived on lib-adc1 to verify it was doing things. Below is the journalctl (sudo journalctl -u keepalived) from the relevant time. Note only the last time (when lib-adc2 was active) does keepalived do a swap. We also noted that the library website was down for approximately 1 second when we restarted lib-adc1 while lib-adc2 was stopped.

Oct 11 11:18:17 lib-adc1 Keepalived_vrrp[975]: Script `chk_nginx_service` now returning 3
Oct 11 11:18:17 lib-adc1 Keepalived_vrrp[975]: VRRP_Script(chk_nginx_service) failed (exited with status 3)
Oct 11 11:18:17 lib-adc1 Keepalived_vrrp[975]: (VI_1) Changing effective priority from 201 to 151
Oct 11 11:18:20 lib-adc1 Keepalived_vrrp[975]: Script `chk_nginx_service` now returning 0
Oct 11 11:18:20 lib-adc1 Keepalived_vrrp[975]: VRRP_Script(chk_nginx_service) succeeded
Oct 11 11:18:20 lib-adc1 Keepalived_vrrp[975]: (VI_1) Changing effective priority from 151 to 201
Oct 11 11:18:53 lib-adc1 Keepalived_vrrp[975]: Script `chk_nginx_service` now returning 3
Oct 11 11:18:53 lib-adc1 Keepalived_vrrp[975]: VRRP_Script(chk_nginx_service) failed (exited with status 3)
Oct 11 11:18:53 lib-adc1 Keepalived_vrrp[975]: (VI_1) Changing effective priority from 201 to 151
Oct 11 11:18:56 lib-adc1 Keepalived_vrrp[975]: Script `chk_nginx_service` now returning 0
Oct 11 11:18:56 lib-adc1 Keepalived_vrrp[975]: VRRP_Script(chk_nginx_service) succeeded
Oct 11 11:18:56 lib-adc1 Keepalived_vrrp[975]: (VI_1) Changing effective priority from 151 to 201
Oct 11 11:21:59 lib-adc1 Keepalived_vrrp[975]: Script `chk_nginx_service` now returning 3
Oct 11 11:21:59 lib-adc1 Keepalived_vrrp[975]: VRRP_Script(chk_nginx_service) failed (exited with status 3)
Oct 11 11:21:59 lib-adc1 Keepalived_vrrp[975]: (VI_1) Changing effective priority from 201 to 151
Oct 11 11:22:02 lib-adc1 Keepalived_vrrp[975]: (VI_1) Master received advert from 128.112.203.144 with higher priority 201, ours 151
Oct 11 11:22:02 lib-adc1 Keepalived_vrrp[975]: (VI_1) Entering BACKUP STATE
Oct 11 11:22:02 lib-adc1 Keepalived_vrrp[975]: Script `chk_nginx_service` now returning 0
Oct 11 11:22:02 lib-adc1 Keepalived_vrrp[975]: VRRP_Script(chk_nginx_service) succeeded
Oct 11 11:22:02 lib-adc1 Keepalived_vrrp[975]: (VI_1) Changing effective priority from 151 to 201

Note Entering BACKUP STATE is lib-adc1 swapping to an active lib-adc2

[carolyncole]

Highlighting the imprtant bits of keepalived journalctl sudo journalctl -u keepalived