Open Liangyuesan opened 2 months ago
There are a number of issues identified here:
You haven't stated what version of keepalived you are using. Please provide the output of keepalived -v.
There are a number of issues identified here:↳
- There is no configuration keyword _notify_ - see line **notify /etc/keepalived/scripts/notify__mysql.sh #set read_only=0**
- _Jun 17 20:42:49 oadb02 Keepalived_vrrp[144285]: (VI1) Entering STATE is strange. It should say what state is being entered, i.e. BACKUP, FAULT or MASTER.
- I can't see any log entry for entering MASTER state, despite the subject of this issue.
- You have multiple _(VI1) received an invalid passwd! entries, but these are not consistent. What is happening that is causing the password that is being sent to change?
You haven't stated what version of keepalived you are using. Please provide the output of keepalived -v.
There are a number of issues identified here:↳
- There is no configuration keyword _notify_ - see line **notify /etc/keepalived/scripts/notify__mysql.sh #set read_only=0**
- _Jun 17 20:42:49 oadb02 Keepalived_vrrp[144285]: (VI1) Entering STATE is strange. It should say what state is being entered, i.e. BACKUP, FAULT or MASTER.
- I can't see any log entry for entering MASTER state, despite the subject of this issue.
- You have multiple _(VI1) received an invalid passwd! entries, but these are not consistent. What is happening that is causing the password that is being sent to change?
You haven't stated what version of keepalived you are using. Please provide the output of keepalived -v.↳
Thank you for your answer The correct conf is as follows global_defs { script_user root enable_script_security } vrrp_script chk_mysql { script "/etc/keepalived/scripts/check_mysql_health.sh" weight -20 interval 1 timeout 30 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 90 advert_int 1 nopreempt authentication { auth_type PASS auth_pass xxx654321 } track_script { chk_mysql } notify_master /etc/keepalived/scripts/notify_master_mysql.sh #set read_only=0 virtual_ipaddress { 172.16.32.51/24 } } the master priority is 100 the keepalived version is Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
3.There are no more records in the system logs 4.the received an invalid passwd maybe The password is longer than 8 characters
There are still some unexplained issues.
There are still some unexplained issues.↳
- Keepalived cannot log "Entering STATE", since the state being entered is always included in the log entry.
- Despite the title of this issue, there are no log entries shown for entering MASTER state.
- The length of the configured password being 9 is not an issue; keepalived simply truncates it. So what is causing the intermittent invalid passwords? Is there some other VRRP instance occasionally coming up on the network? Have you tried configuring the password to be only 8 characters long and seeing what happens then?
Ok, I'll try to change the password to 8 characters first, and then see how the problem happens the tcpdump show tcpdump -i eth0 vrrp -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 14:04:37.939465 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:38.255026 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:38.939519 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:39.255111 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:39.939586 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:40.255197 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:40.939668 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:41.255251 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:41.939771 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:42.255349 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:42.939824 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20
There are still some unexplained issues.↳
- Keepalived cannot log "Entering STATE", since the state being entered is always included in the log entry.
- Despite the title of this issue, there are no log entries shown for entering MASTER state.
- The length of the configured password being 9 is not an issue; keepalived simply truncates it. So what is causing the intermittent invalid passwords? Is there some other VRRP instance occasionally coming up on the network? Have you tried configuring the password to be only 8 characters long and seeing what happens then?
Ok, I'll try to change the password to 8 characters first, and then see how the problem happens the tcpdump show tcpdump -i eth0 vrrp -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 14:04:37.939465 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:38.255026 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:38.939519 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:39.255111 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:39.939586 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:40.255197 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:40.939668 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:41.255251 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:41.939771 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20 14:04:42.255349 IP 172.16.32.b > 224.0.0.18: VRRPv2, Advertisement, vrid 61, prio 100, authtype simple, intvl 1s, length 20 14:04:42.939824 IP 172.16.32.a > 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 100, authtype simple, intvl 1s, length 20↳
hi I stripped the keepalived logs out of the syslogs I noticed something different the switch log as follows, -------> Receive advertisement timeout
Jun 21 16:58:34 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 16:58:35 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 16:58:36 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 16:58:37 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 17:02:51 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 17:02:52 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 17:02:53 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: (VI_1) Receive advertisement timeout Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: (VI_1) Entering MASTER STATE Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: (VI_1) setting VIPs. Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: (VI_1) Sending/queueing gratuitous ARPs on eth0 for 172.16.32.x Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: Sending gratuitous ARP on eth0 for 172.16.32.x Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: Sending gratuitous ARP on eth0 for 172.16.32.x Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: Sending gratuitous ARP on eth0 for 172.16.32.x Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: Sending gratuitous ARP on eth0 for 172.16.32.x Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: Sending gratuitous ARP on eth0 for 172.16.32.x Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: (VI_1) Master received advert from 172.16.32.53 with higher priority 100, ours 90 Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: (VI_1) Entering BACKUP STATE Jun 21 17:02:54 db02 Keepalived_vrrp[207609]: (VI_1) removing VIPs. Jun 21 17:21:15 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd! Jun 21 17:21:16 db02 Keepalived_vrrp[207609]: (VI_1) received an invalid passwd!
the conf is global_defs { script_user root enable_script_security } vrrp_script chk_mysql { script "/etc/keepalived/scripts/check_mysql_health.sh" weight -20 interval 1 timeout 30 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 90 advert_int 1 nopreempt authentication { auth_type PASS auth_pass xxx654321 } track_script { chk_mysql } notify_master /etc/keepalived/scripts/notify_master_mysql.sh #set read_only=0 virtual_ipaddress { 172.16.32.51/24 } } the conf is priority 100 message log print Jun 17 20:42:48 oadb02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd! Jun 17 20:42:49 oadb02 Keepalived_vrrp[144285]: (VI_1) Entering STATE Jun 17 20:42:49 oadb02 Keepalived_vrrp[144285]: (VI_1) received advert from 172.16.32.53 with higher priority 100, ours 90 Jun 17 20:42:49 db02 Keepalived_vrrp[144285]: (VI_1) Entering BACKUP STATE Jun 17 20:42:49 db02 NetworkManager[1002]: [1718628169.4412] policy: set-hostname: current hostname was changed outside NetworkManager: 'db02'
Jun 17 20:42:49 db02 NetworkManager[1002]: [1718628169.4421] policy: set-hostname: current hostname was changed outside NetworkManager: 'db02'
Jun 17 20:50:01 db02 systemd: Started Session 27332 of user root.
Jun 17 20:56:53 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 21:00:01 db02 systemd: Started Session 27333 of user root.
Jun 17 21:00:01 db02 systemd: Started Session 27334 of user root.
Jun 17 21:01:01 db02 systemd: Started Session 27335 of user root.
Jun 17 21:10:01 db02 systemd: Started Session 27336 of user root.
Jun 17 21:20:01 db02 systemd: Started Session 27337 of user root.
Jun 17 21:27:43 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 21:27:44 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 21:27:45 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 21:30:01 db02 systemd: Started Session 27339 of user root.
Jun 17 21:30:01 db02 systemd: Started Session 27338 of user root.
Jun 17 21:40:01 db02 systemd: Started Session 27340 of user root.
Jun 17 21:50:01 db02 systemd: Started Session 27341 of user root.
Jun 17 22:00:01 db02 systemd: Started Session 27343 of user root.
Jun 17 22:00:01 db02 systemd: Started Session 27342 of user root.
Jun 17 22:01:01 db02 systemd: Started Session 27344 of user root.
Jun 17 22:04:23 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:10:01 db02 systemd: Started Session 27345 of user root.
Jun 17 22:10:18 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:10:19 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:20:01 db02 systemd: Started Session 27346 of user root.
Jun 17 22:30:01 db02 systemd: Started Session 27347 of user root.
Jun 17 22:30:01 db02 systemd: Started Session 27348 of user root.
Jun 17 22:40:01 db02 systemd: Started Session 27349 of user root.
Jun 17 22:50:01 db02 systemd: Started Session 27350 of user root.
Jun 17 22:55:23 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:55:24 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:55:25 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:56:40 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:56:41 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:56:42 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:57:56 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:57:57 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:57:58 db02 Keepalived_vrrp[144285]: (VI_1) received an invalid passwd!
Jun 17 22:57:59 db02 Keepalived_vrrp[144285]: (VI_1) Entering STATE
the keepalivbed version is /usr/local/keepalived/sbin/keepalived -v Keepalived v2.2.8 (04/04,2023), git commit v2.2.7-154-g292b299e+
Copyright(C) 2001-2023 Alexandre Cassen, acassen@gmail.com
Built with kernel headers for Linux 3.10.0 Running on Linux 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 Distro: CentOS Linux 7 (Core)
configure options: --prefix=/usr/local/keepalived/
Config options: LVS VRRP VRRP_AUTH VRRP_VMAC OLD_CHKSUM_COMPAT INIT=systemd
System options: VSYSLOG LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTA_VIA IFA_FLAGS NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION LIBIPVS_NETLINK IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE SO_MARK