search

acassen / keepalived

Keepalived
https://www.keepalived.org
GNU General Public License v2.0
3.95k stars 736 forks source link

Cannot change vmac name on conf reload #1798

Closed louis-6wind closed 3 years ago

louis-6wind commented 3 years ago

Describe the bug Changing vmac name does not work on config reload. Works on config restart.

To Reproduce apt install systemd-coredump -y systemctl stop keepalived rm /var/lib/systemd/coredump/* ifconfig ens4 192.168.15.41/28 up cat>/etc/keepalived/keepalived.conf <<\EOF global_defs { router_id router enable_script_security script_user root dynamic_interfaces }

vrrp_sync_group group15 { group { vrrp } }

vrrp_track_file fp_tracker { file /var/run/keepalived/fp-tracker weight 0 }

vrrp_instance vrrp { version 2 state BACKUP interface ens4

use_vmac vrrp

track_file { }

garp_master_delay 5

virtual_router_id 15

priority 200 advert_int 1.0

virtual_ipaddress { 192.168.15.38/28 }

preempt_delay 0 } EOF systemctl restart keepalived ip link show vrrp ip link show vrrpnew sleep 10 sed -e 's|use_vmac vrrp|use_vmac vrrpnew|g' -i /etc/keepalived/keepalived.conf kill -s SIGHUP $(pgrep keepalived | head -n1) #signal for conf reload ip link show vrrp ip link show vrrpnew sleep 10 systemctl restart keepalived ip link show vrrp ip link show vrrpnew coredumpctl | grep -v missing

Observed behavior vmac name is not changed after reload Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: (vrrp) Found matching interface vrrp Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: (vrrp) vmac name mismatch vrrpnew <=> vrrp. changing to vrrp.

Expected behavior vmac name changes after reload

Keepalived version Keepalived v2.1.5 (07/13,2020)

Copyright(C) 2001-2020 Alexandre Cassen, acassen@gmail.com

Built with kernel headers for Linux 4.15.18 Running on Linux 4.15.0-123-generic #126-Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020

configure options: --prefix=/usr --sysconfdir=/etc --with-extra-cflags=-I/usr/include/libnl3 --with-extra-ldflags= --with-extra-libs=-lnl-genl-3 --disable-lvs --with-init=systemd --host=x86_64-linux-gnu host_alias=x86_64-linux-gnu

Config options: VRRP VRRP_AUTH OLD_CHKSUM_COMPAT FIB_ROUTING

System options: PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV4_DEVCONF IPV6_ADVANCED_API RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_OIFNAME RTA_TTL_PROPAGATE IFA_FLAGS IP_MULTICAST_ALL LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA NET_LINUX_IF_H_COLLISION LIBIPTC_LINUX_NET_IF_H_COLLISION VRRP_VMAC VRRP_IPVLAN IFLA_LINK_NETNSID CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH GLOB_BRACE INET6_ADDR_GEN_MODE VRF SO_MARK SCHED_RESET_ON_FORK

Distro (please complete the following information):

Details of any containerisation or hosted service (e.g. AWS) no container

Configuration file: see to reproduce

Notify and track scripts no

System Log entries => process start Dec 02 17:51:20 ha1 systemd[1]: Starting LVS and VRRP High Availability Monitor... Dec 02 17:51:20 ha1 systemd[1]: keepalived.service: Can't open PID file /run/keepalived.pid (yet?) after start: No such file or directory Dec 02 17:51:20 ha1 networkd-dispatcher[704]: WARNING:Unknown index 168 seen, reloading interface list Dec 02 17:51:20 ha1 Keepalived[8416]: Starting Keepalived v2.1.5 (07/13,2020) Dec 02 17:51:20 ha1 systemd-networkd[611]: vrrp: Link UP Dec 02 17:51:20 ha1 Keepalived[8416]: WARNING - keepalived was build for newer Linux 4.15.18, running on Linux 4.15.0-123-generic #126-Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 Dec 02 17:51:20 ha1 systemd-networkd[611]: vrrp: Gained carrier Dec 02 17:51:20 ha1 Keepalived[8416]: Command line: '/usr/sbin/keepalived' '-D' Dec 02 17:51:20 ha1 systemd[1]: Started LVS and VRRP High Availability Monitor. Dec 02 17:51:20 ha1 Keepalived[8416]: Opening file '/etc/keepalived/keepalived.conf'. Dec 02 17:51:20 ha1 systemd-udevd[8430]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. Dec 02 17:51:20 ha1 Keepalived[8417]: NOTICE: setting config option max_auto_priority should result in better keepalived performance Dec 02 17:51:20 ha1 Keepalived[8417]: Starting VRRP child process, pid=8428 Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: Registering Kernel netlink reflector Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: Registering Kernel netlink command channel Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: Opening file '/etc/keepalived/keepalived.conf'. Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: "vrrp_track_file" is deprecated, please use "track_file" Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: Sync group group15 has only 1 virtual router(s) - this probably isn't what you want - removing Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: Assigned address 192.168.15.41 for interface ens4 Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: Assigned address fe80::e56:39ff:fe79:1901 for interface ens4 Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: (vrrp): Success creating VMAC interface vrrp Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: Registering gratuitous ARP shared channel Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: (vrrp) Entering BACKUP STATE (init) Dec 02 17:51:20 ha1 Keepalived_vrrp[8428]: VRRP sockpool: [ifindex(168), family(IPv4), proto(112), fd(11,12)] Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: (vrrp) Receive advertisement timeout Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: (vrrp) Entering MASTER STATE Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: (vrrp) setting VIPs. Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: (vrrp) Sending/queueing gratuitous ARPs on vrrp for 192.168.15.38 Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:23 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:28 ha1 Keepalived_vrrp[8428]: (vrrp) Sending/queueing gratuitous ARPs on vrrp for 192.168.15.38 Dec 02 17:51:28 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:28 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:28 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:28 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38 Dec 02 17:51:28 ha1 Keepalived_vrrp[8428]: Sending gratuitous ARP on vrrp for 192.168.15.38

=> process reload Dec 02 17:51:48 ha1 Keepalived[8417]: Reloading ... Dec 02 17:51:48 ha1 Keepalived[8417]: Opening file '/etc/keepalived/keepalived.conf'. Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: Reloading Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: Opening file '/etc/keepalived/keepalived.conf'. Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: "vrrp_track_file" is deprecated, please use "track_file" Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: Sync group group15 has only 1 virtual router(s) - this probably isn't what you want - removing Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: (vrrp) Found matching interface vrrp Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: (vrrp) vmac name mismatch vrrpnew <=> vrrp. changing to vrrp. Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: Assigned address 192.168.15.41 for interface ens4 Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: Assigned address fe80::e56:39ff:fe79:1901 for interface ens4 Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: (vrrp) setting VIPs. Dec 02 17:51:48 ha1 Keepalived_vrrp[8428]: VRRP sockpool: [ifindex(168), family(IPv4), proto(112), fd(11,12)]

=> process restart Dec 02 17:53:30 ha1 Keepalived[8417]: Stopping Dec 02 17:53:30 ha1 Keepalived_vrrp[8428]: (vrrp) sent 0 priority Dec 02 17:53:30 ha1 Keepalived_vrrp[8428]: (vrrp) removing VIPs. Dec 02 17:53:30 ha1 systemd[1]: Stopping LVS and VRRP High Availability Monitor... Dec 02 17:53:31 ha1 systemd-networkd[611]: vrrp: Link DOWN Dec 02 17:53:31 ha1 systemd-networkd[611]: vrrp: Lost carrier Dec 02 17:53:31 ha1 Keepalived_vrrp[8428]: vmac: Success removing VMAC interface vrrp for vrrp_instance vrrp Dec 02 17:53:31 ha1 systemd[1]: Started Process Core Dump (PID 8459/UID 0). Dec 02 17:53:31 ha1 Keepalived[8417]: Keepalived_vrrp exited due to signal 6 (Aborted) Dec 02 17:53:31 ha1 Keepalived[8417]: CPU usage (self/children) user: 0.000984/0.000000 system: 0.000000/0.068377 Dec 02 17:53:31 ha1 Keepalived[8417]: Stopped Keepalived v2.1.5 (07/13,2020) Dec 02 17:53:31 ha1 systemd[1]: Started Process Core Dump (PID 8461/UID 0). Dec 02 17:53:32 ha1 systemd[1]: keepalived.service: Main process exited, code=dumped, status=6/ABRT Dec 02 17:53:32 ha1 systemd[1]: keepalived.service: Failed with result 'core-dump'. Dec 02 17:53:32 ha1 systemd[1]: Stopped LVS and VRRP High Availability Monitor. Dec 02 17:53:32 ha1 systemd[1]: Starting LVS and VRRP High Availability Monitor... Dec 02 17:53:32 ha1 Keepalived[8463]: Starting Keepalived v2.1.5 (07/13,2020) Dec 02 17:53:32 ha1 Keepalived[8463]: WARNING - keepalived was build for newer Linux 4.15.18, running on Linux 4.15.0-123-generic #126-Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 Dec 02 17:53:32 ha1 Keepalived[8463]: Command line: '/usr/sbin/keepalived' '-D' Dec 02 17:53:32 ha1 Keepalived[8463]: Opening file '/etc/keepalived/keepalived.conf'. Dec 02 17:53:32 ha1 systemd[1]: keepalived.service: Can't open PID file /run/keepalived.pid (yet?) after start: No such file or directory Dec 02 17:53:32 ha1 Keepalived[8474]: NOTICE: setting config option max_auto_priority should result in better keepalived performance Dec 02 17:53:32 ha1 Keepalived[8474]: Starting VRRP child process, pid=8475 Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: Registering Kernel netlink reflector Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: Registering Kernel netlink command channel Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: Opening file '/etc/keepalived/keepalived.conf'. Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: "vrrp_track_file" is deprecated, please use "track_file" Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: Sync group group15 has only 1 virtual router(s) - this probably isn't what you want - removing Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: Assigned address 192.168.15.41 for interface ens4 Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: Assigned address fe80::e56:39ff:fe79:1901 for interface ens4 Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: (vrrp): Success creating VMAC interface vrrpnew Dec 02 17:53:32 ha1 systemd-networkd[611]: vrrpnew: Link UP Dec 02 17:53:32 ha1 systemd-networkd[611]: vrrpnew: Gained carrier Dec 02 17:53:32 ha1 networkd-dispatcher[704]: WARNING:Unknown index 169 seen, reloading interface list Dec 02 17:53:32 ha1 systemd[1]: Started LVS and VRRP High Availability Monitor. Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: Registering gratuitous ARP shared channel Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: (vrrp) Entering BACKUP STATE (init) Dec 02 17:53:32 ha1 systemd-coredump[8460]: Process 8428 (keepalived) of user 0 dumped core.

                                        Stack trace of thread 8428:
                                        #0  0x00007fa514decf47 __GI_raise (libc.so.6)
                                        #1  0x00007fa514dee98b __GI_abort (libc.so.6)
                                        #2  0x00007fa514e37907 __libc_message (libc.so.6)
                                        #3  0x00007fa514e3e97a malloc_printerr (libc.so.6)
                                        #4  0x00007fa514e45f3c munmap_chunk (libc.so.6)
                                        #5  0x000055580a9f927a free_parent_mallocs_exit (keepalived)
                                        #6  0x000055580a9f7a8d vrrp_terminate_phase2 (keepalived)
                                        #7  0x000055580aa066ef start_vrrp_child (keepalived)
                                        #8  0x000055580a9f8d84 start_keepalived (keepalived)
                                        #9  0x000055580aa2f7e8 thread_call (keepalived)
                                        #10 0x000055580a9fa371 keepalived_main (keepalived)
                                        #11 0x00007fa514dcfb97 __libc_start_main (libc.so.6)
                                        #12 0x000055580a9f82aa _start (keepalived)

Dec 02 17:53:32 ha1 Keepalived_vrrp[8475]: VRRP sockpool: [ifindex(169), family(IPv4), proto(112), fd(11,12)] Dec 02 17:53:32 ha1 systemd-udevd[8465]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. Dec 02 17:53:32 ha1 systemd-coredump[8462]: Process 8417 (keepalived) of user 0 dumped core.

                                        Stack trace of thread 8417:
                                        #0  0x00007fa514decf47 __GI_raise (libc.so.6)
                                        #1  0x00007fa514dee98b __GI_abort (libc.so.6)
                                        #2  0x00007fa514e37907 __libc_message (libc.so.6)
                                        #3  0x00007fa514e3e97a malloc_printerr (libc.so.6)
                                        #4  0x00007fa514e45f3c munmap_chunk (libc.so.6)
                                        #5  0x000055580a9f927a free_parent_mallocs_exit (keepalived)
                                        #6  0x000055580a9f9f74 keepalived_main (keepalived)
                                        #7  0x00007fa514dcfb97 __libc_start_main (libc.so.6)
                                        #8  0x000055580a9f82aa _start (keepalived)

Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: (vrrp) Receive advertisement timeout Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: (vrrp) Entering MASTER STATE Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: (vrrp) setting VIPs. Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: (vrrp) Sending/queueing gratuitous ARPs on vrrpnew for 192.168.15.38 Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:35 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:40 ha1 Keepalived_vrrp[8475]: (vrrp) Sending/queueing gratuitous ARPs on vrrpnew for 192.168.15.38 Dec 02 17:53:40 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:40 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:40 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:40 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38 Dec 02 17:53:40 ha1 Keepalived_vrrp[8475]: Sending gratuitous ARP on vrrpnew for 192.168.15.38

Did keepalived coredump? see journalctl

Additional context no

pqarmitage commented 3 years ago

There are two issues here.

The first one is that you can't change the vmac name on a conf reload. That is by design and the log entry (vrrp) vmac name mismatch vrrpnew <=> vrrp. changing to vrrp. is the message relating to not being able to change the name. If you have a real need to change the vmac name, could you please describe the use case.

The second issue is the segfaults. This is very strange since both the parent process and the vrrp process are segfaulting at the same place. The segfault is being triggered from free_parent_mallocs_exit(), which means that either vrrp_pidfile or config_id is invalid. The keepalived command line suggests that both of there should be 0, and free_vrrp_pidfile should be false. Since both processes are experiencing the same segfault, this suggests that the error is occurring before the parent process forks the vrrp child process. Is this segfault reproducable?

louis-6wind commented 3 years ago

The first one is that you can't change the vmac name on a conf reload. That is by design and the log entry (vrrp) vmac name mismatch vrrpnew <=> vrrp. changing to vrrp. is the message relating to not being able to change the name. If you have a real need to change the vmac name, could you please describe the use case.

The use case is we create a new vmac to test and at the end, we want to put a final name without restarting the process

The second issue is the segfaults. This is very strange since both the parent process and the vrrp process are segfaulting at the same place. The segfault is being triggered from free_parent_mallocs_exit(), which means that either vrrp_pidfile or config_id is invalid. The keepalived command line suggests that both of there should be 0, and free_vrrp_pidfile should be false. Since both processes are experiencing the same segfault, this suggests that the error is occurring before the parent process forks the vrrp child process. Is this segfault reproducable?

I am trying to reproduce the issue

pqarmitage commented 3 years ago

Many thanks for you pull request #1801. I have now merge a very minor modification to it as commit a43991d.

I will close this issue now since the reported issue is resolved. If the segfault recurs, I think it should be dealt with as a separate issue.