Use both vxlan and host-gw as flannel backend at the same time

[zq-david-wang]

This is just a Feature Request. (I did not find anyone bring this up before.) I think it is a good idea to combine vxlan and host-gw together to make a backend for a cluster which has several sub-networks: within each sub-networks, packages are routed via host-gw since those hosts are connected together, and vxlan is used for inter-networks package routing.

I used to use vxlan as flannel backend, but it seems to have serious performance degradation within 10Gb network. (My tests show that its performance for single connection is limited by the power of 1 CPU core, in my 2.5GHz x48 cores workstation, its best performance could only reach 70%-80% of total network bandwidth, in bad situation it would only reach 20%, not clear why all cpu load for a single connection is put on a single core though..) There are some suggestions for vxlan performance improvement out there, only that I have not yet found one solution works for me. Then I tried host-gw, it works amazingly well, almost reach the limit of 10Gb network bandwidth. But I have a cluster with two subnet in it, those two subnets are connected by some routers which means that there is no direct L2 connection between those node.

After some thought, I started to make some code changes to test my idea. Well, it works! (I am still testing it.)

(I am just a very new go programmer, even though I would like very much to make contribution, it would take time for me to write acceptable code. )

Expected Behavior

There would be 1 local configuration parameter(say nw.local_network_mask) to determine local sub-network. For example, flannel network 172.10.0.0/16 node1: 192.168.1.2 flannel lease 172.10.2.0/24 local_network_mask=255.255.255.0 node2: 192.168.1.3 flannel lease 172.10.3.0/24 local_network_mask=255.255.255.0 node3: 192.168.2.2 flannel lease 172.10.4.0/24 local_network_mask=255.255.255.0

node1 and node2 are in same local network, node3 is in another network, and they are connected by a router. When new flannel net detected, backend would act like vxlan backend and add local route only the new net's host ip and local host ip are in the same local network. Following is just illustrate the idea:

if ((uint32)(local_ip ^ remote_ip) & nw.local_network_mask ) != 0{ log.V(0).Info("only gw local network, ignore: ", remote_ip.ToIP()) continue }

Thus, route rules for each node would be: node1: 172.10.0.0/16 via flannel.1 172.10.3.0/24 via 192.168.1.3 node2: 172.10.0.0/16 via flannel.1 172.10.2.0/24 via 192.168.1.2 node3: 172.10.0.0/16 via flannel.1 (Since 192.168.1.* has different network id, according to the local_network_mask, no route entry created here.)

Because the route rules for host-gw are stricter than flannel.1 rule, hence host-gw and vxlan should be able to work together.

Current Behavior

Possible Solution

As I mentioned above

Steps to Reproduce (for bugs)

1. 2. 3. 4.

Context

Your Environment

• Flannel version:
• Backend used (e.g. vxlan or udp):
• Etcd version:
• Kubernetes version (if used):
• Operating System and version:
• Link to your project (optional):

[tomdee]

@zq-david-wang thanks for raising this issue. It's something I've been thinking about for a little while and would welcome PRs for it. I'm also hoping to find some time to work on it in the next few months.

It would be similar to the cross-subnet ipip mode in calico - http://docs.projectcalico.org/v2.3/usage/configuration/ip-in-ip

[zq-david-wang]

@tomdee Glad to hear that~!

[tomdee]

Closing as a dupe of #519