calico with eBPF dataplane block keepalived multicast packets

xander-sh commented 3 years ago

We using vanilla kubernetes v1.20.4 with Calico v3.19.1 and keepalived as floating ip for cluster endpoint (kube-api) on vSphere virtual machine. When we enable eBPF dataplane in calico, keeplaived switches to split-brain mode (each masters has virtual ip for cluster endpoint, in normal mode, virtual ip is present only on one master node). Keepalived is configered to use multicast packets for communication between master nodes.

Expected Behavior

Normal work keeplaived with eBPF Calico.

Current Behavior

Keepalived correct work only in unicast mode

Possible Solution

Steps to Reproduce (for bugs)

kubernetes v1.20.4, calico v3.19.1 with eBPF dataplane
keeplaived on master nodes in multicast mode

Context

Each node has one network interface ens192. On normal mode virtual ip (172.16.234.111) only on one master node.

mst03
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b9:1c:8e brd ff:ff:ff:ff:ff:ff
    inet 172.16.234.114/24 brd 172.16.234.255 scope global noprefixroute ens192
       valid_lft forever preferred_lft forever
    inet 172.16.234.111/32 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb9:1c8e/64 scope link
       valid_lft forever preferred_lft forever

In split-brain mode: virtual ip (172.16.234.111) present on each master nodes.

keepalived config

global_defs {
   router_id LVS_DEVEL
}
vrrp_script check_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 3
   weight -2
   fall 10
   rise 2
}
vrrp_instance VI_1 {
    state MASTER
    interface ens192
   mcast_src_ip 172.16.234.112
    virtual_router_id 111
    priority 120
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass h7ESMEFDuQjGuroBj2ZMzO3rZicJOdHJ
    }
    virtual_ipaddress {
        172.16.234.111
    }
    track_script {
       check_apiserver
    }
}

Calico Felix configuration

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
spec:
  bpfEnabled: true
  bpfExternalServiceMode: Tunnel
  bpfLogLevel: ""
  logSeverityScreen: Info
  reportingInterval: 0s
  vxlanEnabled: true

Your Environment

Calico version v3.19.1
Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes v1.20.4
Operating System and version: Oracle Linux 7, kernel 5.4.17-2102.202.5.el7uek.x86_64

caseydavenport commented 3 years ago

@cdtomkins for keepalived config

cdtomkins commented 3 years ago

@tomastigera @caseydavenport I built a lab to test the keepalived config and also to confirm the MACs/IPs.

Here are the findings:

The keepalived.conf works fine as described by @xander-sh (though I removed the check_apiserver functionality)
The resulting MAC for the virtual IP is the real IP of the current master node. This is not related to the issue but @xander-sh you might want to tweak your config to use a virtual MAC as well as a virtual IP as adjacent boxes (particularly firewalls) sometimes get upset about the apparent MAC-hijacking
The VRRP keepalives (that are probably being blocked and causing the split brain) are destined for 01:00:5e:00:00:12 and 224.0.0.18.

chris @ chris-work ~ 
└─515─▶ arp -a | grep -i 10.230
? (10.230.34.9) at 52:54:00:42:98:74 [ether] on mpqemubr0
? (10.230.34.100) at 52:54:00:75:27:5f [ether] on mpqemubr0
? (10.230.34.117) at 52:54:00:75:27:5f [ether] on mpqemubr0

cdtomkins commented 3 years ago

vrrptest2_capture.pcap.txt vrrptest1_capture.pcap.txt

cdtomkins commented 3 years ago

Let me know if there's anything else you need @tomastigera

projectcalico / calico