Use overlay network to transfer traffic from director node to gateway node

nirmata / kube-static-egress-ip

Kubernetes CRD to manage static egress IP addresses for workloads

Apache License 2.0

214 stars 38 forks source link

Use overlay network to transfer traffic from director node to gateway node #38

Open murali-reddy opened 4 years ago

murali-reddy commented 4 years ago

We hit several roadbacks in trying to find a solution that works with CNI's that do direct routing and CNI's that use overlay networks. Also finding a solution that works cross subnet was challenging without use ovelay/tunneling.

It seems a reasonable solution that is agnostinc is to use overlay network to direct traffic from director node to gateway node and same overlay network to send the return traffic back to the node. We get two advantages

solution that is agnostic of CNI used
solution that works across cross subnets or zones

Proposal to revamp the project with overlay network solution. Choice of overlay (VXLAN/IP-in-IP etc) is yet to be decided. Will update this issue as progress is made and will share the decisions.

lyyao09 commented 4 years ago

Is there any plan for support calico's ipip mode? Or any implementation idea? We want to use it in calico's ipip mode.

murali-reddy commented 4 years ago

@lyyao09 Can you please follow the instructions in https://github.com/nirmata/kube-static-egress-ip#installation to try latest master. Now overlay network is used. I have tested with Weave and Kube-router. I dont see a reason why it should not work with Calico. To avoid any conflict with CNI's GRE tunnel based overlay is used so calico in IPIP/VXLAN mode should work too.

lyyao09 commented 4 years ago

@murali-reddy Thank you. I used the latest master, it's works fine with calico's ipip mode.

By the way, has the overlay network become the default implementation? Considering the performance, we hope to use direct route or overlay network according to the actual scenario.

murali-reddy commented 4 years ago

thanks @lyyao09 for testing and confirming it works with calico IPIP mode.

Problem has been to come up with a solution that works agnostic to any CNI. This would be default approach.

Considering the performance, we hope to use direct route or overlay network according to the actual scenario.

With direct routing while it works in single subnet/zone, we will run into problems when cluster is multi subnet/zone. But we will revisit to provide a override mechanism to use direct routing where it can be used.

lyyao09 commented 4 years ago

Okay, looking forward to going well.

WeAreHadock commented 4 years ago

@lyyao09 Like you with calico's ipip mode, i follow instructions and everything seems to run but it does not work (we go out with the node ip). without networker knowledge, i can't diagnose too much. Could you tell us if you had to make any adjustments on documentation or code? would you have any "simple" deployment to provide for us as example?

@murali-reddy Could you please confirm that egressIP must be an unused address on my network (and not a known node IP or a keepalived)? and, can my cluster be on different IP subnetworks to work fine with kube-statuc-egress?

thanks a lot

lyyao09 commented 4 years ago

@WeAreHadock Yes, I made some adjustments on code. Since we already have a vip in my cluster, I disabled the code for configuring egressIP on onde.