[WIP] HA gateway

[murali-reddy]

Refactor current code to support below:

• automate the selection of gateway (currently annotations are used to select gateway)
• new deployment running > 1 replicas, use leader election to run a manger that select a gateway from set of nodes in the cluster
• in case of node failure re-select a different node to act as a gateway
• static-egress-ip controller running on each node do necessary modification on the node to act as a gateway or forwarder depending on type of the node

[murali-reddy]

Gateway manager part is fully functional now. Please follow below step to try out.

On any kubernetes cluster with more than 3 nodes you can try below steps:

• install CRD kubectl apply -f https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-crd.yaml
• install necessary RBAC kubectl apply -f https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-rbac.yaml
• install gateway manager kubectl apply -f https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-gateway-manager.yaml

Now at this point static-egress-gateway-manager components performs following on any applied staticegressip custom resources

• pick gateway for a staticegressip resource and update the status object with Gateway field
• in case selected gateway node dies it automatically detects and chooses new gateway node and updates the status of staticegressip custom resource/

static-egress-gateway-manager itslef runs as HA mode with only one instance at a time actin as leader.

Please note that this steps only for testing logic to choose gateway for static egress IP and detect node failures and switch over to a different gateway.

For actual data-path configurations and routing changes to director and gateway you need to install kubectl apply -f https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-controller.yaml

[murali-reddy]

Updating instruction for testing end-to-end functionality:

• install CRD kubectl apply -f https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-crd.yaml
• install necessary RBAC kubectl apply -f https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-rbac.yaml
• install gateway manager kubectl apply -f https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-gateway-manager.yaml
• install controller deamonset https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-controller.yaml

At this point you can install a customeresource object. For e.g. please take a look at https://raw.githubusercontent.com/nirmata/kube-static-egress-ip/ha/config/static-egressip-example.yaml

you need to ensure service mentioned under service-name must exist.

Please note that this only works with direct-routing CNI's. For instance use Flannel with host-gw.

I am working making functionality work with overlay CNI's e.g. Flannel VXLAN and Weave.

[uablrek]

If kube-static-egress-ip is upgraded with a rolling-upgrade (but the nodes are not re-booted) then it would be horrible if the active gw hops to the next, and next node during the upgrade. When a new gw becomes active external traffic must be re-routed to it (how?) and all existing connections are lost.

Will is be possible to tune the leaderelection in a very "defensive" way. I.e wait for the active gw to come back rather than immediately move to another node?

[murali-reddy]

@uablrek thanks for taking interest in this project.

it would be horrible if the active gw hops to the next, and next node during the upgrade

Right now thinking (implementation in this PR) is to rely upon the kubelet + kube-controller-manager notion of node readiness. When a node acting as a gateway becomes unready, a new node is selected from the set of ready nodes.

[murali-reddy]

Merging PR. I will do follow-up PR to update the documentation.