search

polycube-network / polycube

eBPF/XDP-based software framework for fast network services running in the Linux kernel.
Apache License 2.0
507 stars 102 forks source link

[BUG] (pcn-k8s) Services don't work when pods are on different nodes #201

Open asimpleidea opened 5 years ago

asimpleidea commented 5 years ago

Describe the bug

As per title, if the pods are on different nodes, they cannot be accessed if called by their service.

To Reproduce

For simplicity, let's suppose you have two nodes: so, a master and a worker. Deploy a test pod that will be scheduled on worker: kubectl run apiserver --image=nginx --labels app=apiserver --expose --port 80

Wait until it is running and get its podIP and its service's clusterIP.

Now, get inside a pod that is running on the same node as the pod just deployed, i.e. polycube-worker, and try to access the pod:

kubectl exec -it polycube-worker -n kube-system -- /bin/bash
# Query by its ip
wget -qO- podIP
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
# ... It works

# Query by its service's clusterIP
wget -qO- clusterIP
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
# ... It works
exit

Now, do the same on a pod running in a different node, i.e. polycube-master:

kubectl exec -it polycube-master -n kube-system -- /bin/bash
# Query by its ip
wget -qO- podIP
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
# ... It works

# Query by its service's clusterIP
wget -qO- clusterIP
# ... It hangs
exit

Expected behavior

The pod should be accessible and the following message should be displayed if performing the steps above (of course here the message is cut):

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
...

Please tell us about your environment:

1. OS details: 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
  1. Kernel details: 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  2. Polycube Version: 
    polycubectl:
version: v0.9.0-rc1+ [git: (branch/commit): master/7adff186]
polycubed:
version: v0.1.0-beta+ [git: (branch/commit): master/cc34fabb]
  3. Polycube Log: 
    [2019-01-18 18:21:44.902] [polycubed] [info] configuration parameters: 
[2019-01-18 18:21:44.903] [polycubed] [info]  loglevel: debug
[2019-01-18 18:21:44.903] [polycubed] [info]  daemon: false
[2019-01-18 18:21:44.903] [polycubed] [info]  pidfile: /var/run/polycube.pid
[2019-01-18 18:21:44.903] [polycubed] [info]  port: 9000
[2019-01-18 18:21:44.903] [polycubed] [info]  addr: localhost
[2019-01-18 18:21:44.903] [polycubed] [info]  logfile: /var/log/polycube/polycubed.log
[2019-01-18 18:21:44.903] [polycubed] [info]  root-only: false
[2019-01-18 18:21:44.903] [polycubed] [info] polycubed starting...
[2019-01-18 18:21:44.903] [polycubed] [info] version v0.9.0-rc1+ [git: (branch/commit): master/7adff186]
[2019-01-18 18:21:44.903] [polycubed] [debug] removing root-only iptables rules
[2019-01-18 18:21:45.459] [polycubed] [critical] Error starting polycube: --- Virtual interface pcn_tc_cp is already up.
up() Operation not permitted.

    Additional context

acloudiator commented 5 years ago

Thanks @SunSince90 for reporting this issue. If not mistaken, this was working. We surely needs to identify it this is a bug or an environment setup issue.

goldenrye commented 5 years ago

I can access the service via cluster IP from pod (with IP in the range --pod-network-cidr) in the remote host. Accessing service from host or pcn-k8s pod (it is host IP) seems not covered in pcn_k8s design and I am not sure whether we should support or not -- I didn't see the need because if the service is needed to access from outside the cluster, pcn_k8s can support node port.

mauriciovasquezbernal commented 5 years ago

host to ClusterIP case is already supported (besides any new bug I am not aware of). In this case the packet is processed by kube-proxy that performs the load balancing and changes the destination IP to the one of the selected pod, after it the packet is picked by polycube and forwarded as usual.

goldenrye commented 5 years ago

you are right host to clusterIP case is taken care by Kube-proxy, but based on my observation since the request packet use the host IP as its source IP, when packet goes through the path: host stack->vxlan tunnel->physical NIC->remote vxlan tunnel->pod, the packet will be dropped by kernel due to RP filter since the routing points to different source interface.

goldenrye commented 5 years ago

This is more like a routing issue of Kube-proxy, when wget uses the VXLan interface address as source IP of requesting packet, the packet can go through; on the other hand if wget uses the physical interface IP to build the request packet, problem will occur.

A workaround is to use option "--bind-address" to bind to the VXLan interface when send request.