search

acassen / keepalived

Keepalived
https://www.keepalived.org
GNU General Public License v2.0
4k stars 737 forks source link

IPv6 virtual addresses cause segfault #406

Closed ydahhrk closed 8 years ago

ydahhrk commented 8 years ago

keepalived.conf (stripped down as much as possible):

vrrp_instance VI_1 {
    interface eth0
    virtual_ipaddress {
        fe80::0102:0304
        2001:db8::1
    }

    virtual_router_id 64
}

Is this reasonable?

keepalived -lDn:

Starting Healthcheck child process, pid=4954
Starting VRRP child process, pid=4955
Initializing ipvs 2.6
Netlink reflector reports IP fe80::a00:27ff:fe62:10d6 added
Netlink reflector reports IP fe80::a00:27ff:fe62:10d6 added
Netlink reflector reports IP fe80::a00:27ff:feca:76e6 added
Netlink reflector reports IP fe80::a00:27ff:fe44:4f88 added
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Opening file '/etc/keepalived/keepalived.conf'.
Netlink reflector reports IP fe80::a00:27ff:feca:76e6 added
Netlink reflector reports IP fe80::a00:27ff:fe44:4f88 added
Registering Kernel netlink reflector
Registering Kernel netlink command channel
Registering gratuitous ARP shared channel
Opening file '/etc/keepalived/keepalived.conf'.
VRRP_Instance(VI_1) removing protocol VIPs.
VRRP_Instance(VI_1) removing protocol iptable drop rule
Using LinkWatch kernel netlink reflector...
pid 4955 exited due to segmentation fault (SIGSEGV).
  Please report a bug at https://github.com/acassen/keepalived/issues
  and include this log from when keepalived started, what happened
  immediately before the crash, and your configuration file.
VRRP child process(4955) died: Respawning

It keeps segfaulting and respawning forever.

IPv4 virtual addresses start the daemon just fine.

Running keepalived 1.2.23.

pqarmitage commented 8 years ago

Your config looks reasonable, and it works for me.

If you can do the following we should be able to find where the segfault is occurring.

First, the output of keepalived -v will tell us what options keepalived has been built with.

Build a version of keepalived with symbols - make debug Now run that version as follows: bin/keepalived -lDn -R -m # R stops respawning and -m ensures it will create a core dump

There should now be a file /core.PID, so run the following:

gdb bin/keepalived /core.PID
gdb> bt

That should give you a backtrace of the core dump. If you can post that here, we can try and work out what is happening.

ydahhrk commented 8 years ago

keepalived -v:

Keepalived v1.2.23 (08/21,2016)

Copyright (C) 2001-2016 Alexandre Cassen, <acassen@gmail.com>

Build options: KRNL_2_6 WITH_LVS HAVE_IPVS_SYNCD WITH_VRRP HAVE_VRRP_VMAC WITHOUT_ADDR_GEN_MODE WITHOUT_SNMP WITHOUT_SNMP_KEEPALIVED WITHOUT_SNMP_CHECKER WITHOUT_SNMP_RFC WITHOUT_SNMP_RFCV2 WITHOUT_SNMP_RFCV3 LIBIPVS_USE_NL WITHOUT_LIBNL WITH_VRRP_AUTH WITH_SO_MARK HAVE_LIBIPTC WITHOUT_LIBIPSET WITHOUT_IPV4_DEVCONF WITHOUT_IF_H_LINK_H_COLLISION WITHOUT_LINUX_NET_IF_H_COLLISION HAVE_SOCK_NONBLOCK HAVE_SOCK_CLOEXEC HAVE_FIB_ROUTING NO_MEM_CHECK NO_MEM_CHECK_LOG

bt:

#0  0xb7503919 in ?? () from /lib/libip6tc.so.0
#1  0xb75052f7 in ?? () from /lib/libip6tc.so.0
#2  0x0806339c in ip6tables_process_entry (handle=<optimized out>, 
    chain_name=<optimized out>, rulenum=<optimized out>, rulenum@entry=-1, 
    target_name=<optimized out>, src_ip_address=<optimized out>, 
    src_ip_address@entry=0x0, dst_ip_address=<optimized out>, 
    dst_ip_address@entry=0x895a650, in_iface=<optimized out>, 
    in_iface@entry=0x8951200 "eth0", out_iface=<optimized out>, 
    out_iface@entry=0x0, protocol=<optimized out>, protocol@entry=58, 
    type=<optimized out>, type@entry=135, cmd=<optimized out>, cmd@entry=0)
    at vrrp_iptables_calls.c:304
#3  0x0805c42d in iptables_entry (h=h@entry=0x89594d8, 
    chain_name=<optimized out>, 
    target_name=target_name@entry=0x8076b54 "ACCEPT", 
    src_ip_address=src_ip_address@entry=0x0, 
    dst_ip_address=dst_ip_address@entry=0x895a650, 
    in_iface=in_iface@entry=0x8951200 "eth0", out_iface=out_iface@entry=0x0, 
    protocol=protocol@entry=58, type=type@entry=135, cmd=cmd@entry=0, 
    rulenum=-1) at vrrp_iptables.c:230
#4  0x0805c63c in handle_iptable_rule_to_NA (h=0x89594d8, h@entry=0x895a650, 
    ifname=0x8951200 "eth0", cmd=0, ipaddress=0x895a650) at vrrp_iptables.c:245
#5  handle_iptable_rule_to_vip (ipaddress=0x895a650, cmd=cmd@entry=0, 
    ifname=ifname@entry=0x8951200 "eth0", h=h@entry=0x89594d8)
    at vrrp_iptables.c:295
#6  0x08061ef2 in handle_iptable_rule_to_iplist (h=h@entry=0x89594d8, 
    ip_list=0x895a638, cmd=cmd@entry=0, ifname=0x8951200 "eth0", 
    force=force@entry=true) at vrrp_ipaddress.c:299
#7  0x08065b98 in vrrp_handle_accept_mode (vrrp=0x8959ac8, cmd=0, force=true)
    at vrrp.c:127
#8  0x08066f37 in vrrp_restore_interface (vrrp=vrrp@entry=0x8959ac8, 
    advF=advF@entry=false, force=force@entry=true) at vrrp.c:1310
#9  0x080688c9 in vrrp_complete_instance (vrrp=0x8959ac8) at vrrp.c:2238
#10 vrrp_complete_init () at vrrp.c:2278
#11 0x0805e7f2 in start_vrrp () at vrrp_daemon.c:222
#12 0x0805ec6d in start_vrrp_child () at vrrp_daemon.c:473
#13 0x0804ae30 in start_keepalived () at main.c:95
#14 main (argc=4, argv=0xbfad1c04) at main.c:506

I take it this means that there's something wrong with libip6tc.so, rather?

pqarmitage commented 8 years ago

The difference between your build of keepalived and mine is that I am using libipset, whereas you aren't, and so you are using ip6tables to block packets for non accept mode, whereas I am using ipsets.

I have built keepalived without libipset, and I still don't get the segfault problem you are. I am seeing some strange behaviour though; I am unable to delete the ip6tables entries added by keepalived using the ip6tables command with matching parameters. I can add a rule that looks identical, delete that rule but not delete the keepalived added entry.

For example, I have an entry in chain INPUT: 29 0 0 ACCEPT icmpv6 wlan0 * ::/0 fe80::102:304/128 ipv6-icmptype 135 and when I execute ip6tables -D INPUT -p icmpv6 -i wlan0 -d fe80::102:304/128 --icmpv6-type 135 -j ACCEPT I get the error ip6tables: Bad rule (does a matching rule exist in that chain?). This suggests that there is something wrong with the way the rule is being added by keepalived, and that might cause the segfault you are experiencing, whereas for me the rule gets added but with something strange about it.

The segfault is occuring from line 304 of vrrp_iptables_calls.c: res = ip6tc_delete_entry ( chain, fw, matchmask, handle); This is being called at startup to remove any residual entries that may have been left over from a previous run.

As a workaround to the problem, are you in a position to install the ipset development libraries, then rerunning configure and building keepalived again? Or you could run configure with the --without-libiptc option. Alternatively add the keyword 'accept' to each IPv6 vrrp_instance. Neither of these fixes the problem, but it may enable things to work for you until we can fix the problem.

In general, using libipset is the most efficient, and option --without-libiptc is the least efficient.

In order to help me try and find a system which exhibits the segfault you are experiencing, could you please provide the following information: Linux version, iptables version, and what distro you are using.

ydahhrk commented 8 years ago

As a workaround to the problem, are you in a position to install the ipset development libraries, then rerunning configure and building keepalived again? Or you could run configure with the --without-libiptc option. Alternatively add the keyword 'accept' to each IPv6 vrrp_instance. Neither of these fixes the problem, but it may enable things to work for you until we can fix the problem.

Ok, let's see.

In order to help me try and find a system which exhibits the segfault you are experiencing, could you please provide the following information: Linux version, iptables version, and what distro you are using.

$ uname -a
Linux al-VirtualBox 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:31:42 UTC 2014 i686 i686 i686 GNU/Linux
$ ip6tables --version
iptables v1.4.21

Ubuntu 14.04

ydahhrk commented 8 years ago

Ok, removing libiptc made it work all the way to the end. (I don't mind the performance; I'm just testing for the moment.)

I also made another virtual machine from scratch, as similar to the original as possible, and the segmentation fault did not show up there. I suppose the old VM has had one kernel panic too many and is a little touched in the head now. I should replace it; you will likely not be able to reproduce the problem.

This issue is fixed as far as I'm concerned, but I can leave it open if you want a reminder of the iptables rules mismatch you found. You can close it instead, right?

Thank you very much for your time and sorry for the mess :/

pqarmitage commented 8 years ago

Commit a40aeb8 resolves the issue of malformed iptables/ip6tables entries which cannot be deleted by iptables commands.