A NAT Gateway sits between your DOKS cluster and public Internet. Main role is to control egress traffic, that is traffic going outside your DOKS cluster. Because of the context, we can call it an Egress Gateway as well.
Another benefit of using an egress gateway is that all traffic exiting your cluster is routed through a single public IP, thus making ACLs management easier for external services that require it.
Implementation
Main idea is to have a dedicated droplet acting as an egress gateway for our DOKS cluster. Then, static routes are added to the DOKS worker nodes which tell Linux to route or not traffic through the egress gateway based on destination address. On the other hand, we can have a list of public CIDRs, and route all outbound traffic via the egress gateway (some limitations apply, though).
There are two ways to approach this from a Kubernetes point of view:
Building a DaemonSet embedding the required logic to handle Linux routing tables (either via a script and ip route commands, or programatically via the Pyroute2 Python package). Having a DaemonSet running, we can control routing tables on each of the DOKS cluster nodes. The configuration part is kept using a ConfigMap.
Building a Kubernetes Operator. This is a more advanced approach, but not necessarily complicated as people may think. Main advantage here is that we control the flow via CRDs, which users are already familiar with. Then, we let the Kubernetes control loop take care of the rest and maintain the desired state for the Linux routing table on each node. The operator can be written using Python using the Kopf framework and Pyroute2.
The latter approach is chosen because it offers more flexibility and control over the process. We implemented a custom static routes controller to achieve this goal, called k8s-staticroute-operator.
Background
A NAT Gateway sits between your DOKS cluster and public Internet. Main role is to control egress traffic, that is traffic going outside your DOKS cluster. Because of the context, we can call it an Egress Gateway as well.
Another benefit of using an egress gateway is that all traffic exiting your cluster is routed through a single public IP, thus making ACLs management easier for external services that require it.
Implementation
Main idea is to have a dedicated droplet acting as an egress gateway for our DOKS cluster. Then, static routes are added to the DOKS worker nodes which tell Linux to route or not traffic through the egress gateway based on destination address. On the other hand, we can have a list of public CIDRs, and route all outbound traffic via the egress gateway (some limitations apply, though).
There are two ways to approach this from a Kubernetes point of view:
The latter approach is chosen because it offers more flexibility and control over the process. We implemented a custom static routes controller to achieve this goal, called k8s-staticroute-operator.