search

cybozu-go / coil

CNI plugin for Kubernetes designed for scalability and extensibility
Apache License 2.0
158 stars 18 forks source link

Enhance the graceful termination for Egress NAT #220

Closed ysksuzuki closed 1 year ago

ysksuzuki commented 2 years ago

What

Rolling restart of Egress NAT pods causes a brief outage

  1. Set deletionTimestamp to a nat pod, and it becomes terminating
  2. kubelet sends SIGTERM to the container process <- brief outage from here until cilium removes the backend
  3. NAT pod gets removed <- cilium removed the backend, and send packets to a new backend

How

  1. Set deletionTimestamp to a nat pod, and it becomes terminating
  2. Sleep for a while during its preStop hook <- cilium notices that the backend becomes inactive and selects a new backend
  3. kubelet sends SIGTERM to the container process <- no outage since cilium has already selected a new backend
  4. NAT pod gets removed

Cilium selects a new backend if the client hits the same old tuple for syn packets, but it doesn't consider UDP packets. So we need to send a PR for it. https://github.com/cilium/cilium/pull/20407

Checklist

ymmt2005 commented 2 years ago

@ysksuzuki I think this can be done without preStop hook by holding SIGTERM processing for a while in coil-egress.

ysksuzuki commented 1 year ago

Selecting a new backend for UDP packets breaks the graceful termination since it's not possible to check whether a received UDP packet is new or existing one in the current implementation. So we will go for another approach.