acassen / keepalived

Keepalived
https://www.keepalived.org
GNU General Public License v2.0
3.98k stars 735 forks source link

VRRPv3 IPv6: persistent FAULT state when updating VMAC with a non-existent VMAC base interface #2280

Open louis-6wind opened 1 year ago

louis-6wind commented 1 year ago

Describe the bug VRRPv3 IPv6 remains in a persistent FAULT state after setting an non-existent VMAC base interface. When the base interface appears, the problem is not solved. Base interface is never updated on the vmac.

To Reproduce

On a standalone VM machine

# Apply the following commands
killall keepalived
sleep 1
ip link set iface5 down
ip link set iface5 name ens5
ip link set ens5 down

ip -6 addr add  fd00:100::3/64 dev ens4
ip addr add 10.0.0.1/24 dev ens4
ip link set dev ens4 up
sleep 2
cat>/etc/keepalived/keepalived.conf <<\EOF
global_defs
{
router_id router
dynamic_interfaces
}

vrrp_sync_group group15 {
group {
vrrp
}
}

vrrp_instance vrrp {
version 3
state BACKUP
interface ens4

use_vmac vrrp

garp_master_delay 5

virtual_router_id 15

priority 200
advert_int 1.0

virtual_ipaddress {
fd00:100::1/64
}

preempt_delay 0
}
EOF
keepalived -D
sleep 5
journalctl _PID=$(pgrep keepalived | tail -n1) | cat
sed -e 's|ens4|iface5|g' -i /etc/keepalived/keepalived.conf
killall -s SIGHUP keepalived
sleep 4
ip -br l
journalctl _PID=$(pgrep keepalived | tail -n1) | cat
ip link set ens5 name iface5 up
sleep 4
journalctl _PID=$(pgrep keepalived | tail -n1) | cat
ip -br l

Expected behavior

Instance in MASTER state. VMAC updated with the correct base interface

Keepalived version

Keepalived v2.2.7 (04/02,2023), git commit v2.2.7-148-g58be65ee

Copyright(C) 2001-2023 Alexandre Cassen, <acassen@gmail.com>

Built with kernel headers for Linux 4.15.18
Running on Linux 5.4.0-135-generic #152~18.04.2-Ubuntu SMP Tue Nov 29 08:23:49 UTC 2022
Distro: Ubuntu 18.04.1 LTS

configure options: CFLAGS=-g -O0 --prefix=/usr --sysconfdir=/etc --with-extra-cflags=-I/usr/include/libnl3 --with-extra-ldflags= --disable-lvs --with-init=systemd --host=x86_64-linux-gnu host_alias=x86_64-linux-gnu

Config options:  NFTABLES VRRP VRRP_AUTH VRRP_VMAC OLD_CHKSUM_COMPAT INIT=systemd SYSTEMD_NOTIFY

System options:  VSYSLOG MEMFD_CREATE IPV4_DEVCONF RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA RTA_TTL_PROPAGATE IFA_FLAGS LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA IPTABLES NET_LINUX_IF_H_COLLISION VRRP_IPVLAN IFLA_LINK_NETNSID GLOB_BRACE GLOB_ALTDIRFUNC INET6_ADDR_GEN_MODE VRF SO_MARK

Distro (please complete the following information):

Details of any containerisation or hosted service (e.g. AWS)

NA

Configuration file: See above Notify and track scripts NA

System Log entries

Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: Registering Kernel netlink reflector
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: Registering Kernel netlink command channel
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: Sync group group15 has only 1 virtual router(s) - this probably isn't what you want
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) the first IPv6 VIP address should be link local
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: use_vmac or no_accept/strict specified, but no firewall configured - using nftables
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: Assigned address 10.0.0.1 for interface ens4
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: Assigned address fe80::dced:1ff:fe70:f533 for interface ens4
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp): Success creating VMAC interface vrrp
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: Assigned address fe80::dced:1ff:fe70:f533 for interface vrrp
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: Registering gratuitous NDISC shared channel
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) Entering BACKUP STATE (init)
Apr 04 18:43:01 ubuntu1804hwe Keepalived_vrrp[21742]: VRRP sockpool: [ifindex(164), family(IPv6), proto(112), fd(13,14) multicast, address(ff02::12)]
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) Receive advertisement timeout
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) Entering MASTER STATE
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) setting VIPs.
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) Sending/queueing Unsolicited Neighbour Adverts on vrrp for fd00:100::1
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: Sending unsolicited Neighbour Advert on vrrp for fd00:100::1
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: Sending unsolicited Neighbour Advert on vrrp for fd00:100::1
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: Sending unsolicited Neighbour Advert on vrrp for fd00:100::1
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: Sending unsolicited Neighbour Advert on vrrp for fd00:100::1
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: Sending unsolicited Neighbour Advert on vrrp for fd00:100::1
Apr 04 18:43:04 ubuntu1804hwe Keepalived_vrrp[21742]: VRRP_Group(group15) Syncing instances to MASTER state
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: Reloading
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: Configuration specifies interface iface5 which doesn't currently exist - will use if created
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: Sync group group15 has only 1 virtual router(s) - this probably isn't what you want
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) the first IPv6 VIP address should be link local
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: use_vmac or no_accept/strict specified, but no firewall configured - using nftables
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) VMAC vrrp already exists but is incompatible. It will be deleted/updated
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: Assigned address fe80::dced:1ff:fe70:f533 for interface vrrp
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp): entering FAULT state (interface iface5 down)
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) entering FAULT state (no IPv6 address for interface)
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: VRRP_Group(group15): Syncing vrrp to FAULT state
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) entering FAULT state
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) sent 0 priority
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) removing VIPs.
Apr 04 18:43:06 ubuntu1804hwe Keepalived_vrrp[21742]: VRRP sockpool: [ifindex(164), family(IPv6), proto(112), fd(13,14) multicast, address(ff02::12)]
Apr 04 18:43:10 ubuntu1804hwe Keepalived_vrrp[21742]: Interface name has changed from ens5 to iface5
Apr 04 18:43:10 ubuntu1804hwe Keepalived_vrrp[21742]: Interface iface5 added
Apr 04 18:43:10 ubuntu1804hwe Keepalived_vrrp[21742]: (vrrp) interface iface5 is down
Apr 04 18:43:10 ubuntu1804hwe Keepalived_vrrp[21742]: Netlink reports iface5 up
Apr 04 18:43:12 ubuntu1804hwe Keepalived_vrrp[21742]: Assigned address fe80::dced:1ff:fe70:f534 for interface iface5
Apr 04 18:43:12 ubuntu1804hwe Keepalived_vrrp[21742]: Assigned address fe80::dced:1ff:fe70:f534 for interface vrrp
+ ip -br l
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
mgmt0            UP             de:ad:de:01:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP> 
ens4             UP             de:ed:01:70:f5:33 <BROADCAST,MULTICAST,UP,LOWER_UP> 
iface5           UP             de:ed:01:70:f5:34 <BROADCAST,MULTICAST,UP,LOWER_UP> 
ens6             DOWN           de:ed:01:70:f5:35 <BROADCAST,MULTICAST> 
ens7             DOWN           de:ed:01:70:f5:35 <BROADCAST,MULTICAST> 
vrrp@ens4        UP             00:00:5e:00:02:0f <BROADCAST,MULTICAST,UP,LOWER_UP> 

Did keepalived coredump? No

louis-6wind commented 1 year ago

To fix the issue, patches https://github.com/acassen/keepalived/pull/2277 and https://github.com/acassen/keepalived/pull/2281 are needed.

louis-6wind commented 1 year ago

A more complicated scenario works with the patches

# Apply the following commands
killall keepalived
sleep 1
ip link set iface5 down
ip link set iface5 name ens5
ip link set ens5 down

ip -6 addr add  fd00:100::3/64 dev ens4
ip addr add 10.0.0.1/24 dev ens4
ip link set dev ens4 up
sleep 2
cat>/etc/keepalived/keepalived.conf <<\EOF
global_defs
{
router_id router
dynamic_interfaces
}

vrrp_sync_group group15 {
group {
vrrp
}
}

vrrp_instance vrrp {
version 3
state BACKUP
interface ens4

use_vmac vrrp

garp_master_delay 5

virtual_router_id 15

priority 200
advert_int 1.0

virtual_ipaddress {
fd00:100::1/64
}

preempt_delay 0
}
EOF
keepalived -D
sleep 5
journalctl _PID=$(pgrep keepalived | tail -n1) | cat
sed -e 's|ens4|iface6|g' -i /etc/keepalived/keepalived.conf
killall -s SIGHUP keepalived
sleep 4
ip -br l
journalctl _PID=$(pgrep keepalived | tail -n1) | cat
sed -e 's|iface6|iface5|g' -i /etc/keepalived/keepalived.conf
killall -s SIGHUP keepalived
sleep 4
ip -br l

ip link set ens5 name iface5 up
sleep 4
journalctl _PID=$(pgrep keepalived | tail -n1) | cat
ip -br l