search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.95k stars 1.32k forks source link

run eBPF + DSR on top of Wireguard ( Layer 3 VPN ) #4326

Open Bobonium opened 3 years ago

Bobonium commented 3 years ago

I've got nodes in two differnet datacenters that are not directly connected, where not all nodes have a public IP. To run a single Cluster between the datacenters it is running on top of a wireguard connection. Not the Calico wireguard feature, but a standalone wireguard VPN connecting all the nodes Calico itself is working without problems, but as soon as the eBPF layer is enabled all traffic is dropped.

Expected Behavior

Running Calico with eBPF + DSR on top of an existing wireguard Network should work

Current Behavior

All traffic is dropped in the wireguard interface, as soon as eBPF is enabled

Possible Solution

After talking to @fasaxc in slack, he told me that the reason for this behavior lies in the fact that the BPF program expects the ethernet header, which does not exist on the wireguard tunnel, as it's a layer 3 device. The program needs to be made aware of that to parse the packets correctly

Steps to Reproduce (for bugs)

  1. Setup Wireguard Network
  2. Deploy Cluster on top of Wireguard Network
  3. Deploy Calico
  4. Enable eBPF
  5. adjust felix config bpfDataIfacePattern to include wireguard interface
  6. all traffic on the wireguard interface is dropped

Context

Two datacenters that do not have a direct connection to each other should run a single Kubernetes Cluster, to allow traffic between all nodes a VPN layer is necessary

Your Environment

kerberjg commented 3 years ago

@Bobonium were you able to get Wireguard + Calico running without eBPF?

Bobonium commented 3 years ago

Hey @kerberjg ,

yes that is what I'm running for 2 or 3 years now without any problems. It just doesn't work with eBPF currently, since the BPF program that belongs to calico does not support being run behind the VPN. I also never found the time to look at the necessary code changes myself

kerberjg commented 3 years ago

@Bobonium That's great! We've been trying to achieve a similar setup for the past few weeks, but we've been running into some issues. I'm really interested in hearing about your experience with it, do you think we could have a chat?

Bobonium commented 3 years ago

@kerberjg since this is not really related to the issue, feel free to send me an E-Mail to the address in my github bio. If I find some time I'll have a quick look at it, but I can't make you any promises on when that exactly might be. I'd need to know how exactly you deploy kubernetes (kubeadm/kubespray/rancher/???) and which Kubernetes distribution you're using. I'd also need to know how you deploy calico and the configuration you use for it, as well as how your wireguard setup looks like at least in a rough description.

kerberjg commented 3 years ago

Thanks! I sent you an email :)