ctrox / zeropod

pod that scales down to zero
Apache License 2.0
74 stars 6 forks source link

feat: implement bpf-steered activator #2

Closed ctrox closed 10 months ago

ctrox commented 10 months ago

The old activator had several problems:

While the new activator is not fully realised in eBPF, it's way more reliable as we can simply steer traffic without any interruptions just with a few maps. Essentially activation now works like this:

  1. container is in checkpointed state.
  2. incoming packet destined to container.
  3. eBPF program redirects packet to userspace TCP proxy listening on random free port.
  4. proxy accepts TCP session and triggers restore of container.
  5. proxy connects to container as soon as it's running.
  6. proxy shuffles data back and forth for this TCP session and all other connections that were established while the container was restoring.
  7. write to eBPF map to indicate it no longer needs to redirect to proxy.
  8. traffic flows to container directly as usual without going through the proxy for as long as it's alive.
  9. on checkpoint the redirect is enabled again.

It still only needs to proxy the requests during restore while having a more reliable activator that never drops a packet. The current implementation is using TC as it allows to modify ingress and egress packets. A full eBPF solution has been experimented with but the main issue is that we need to "hold back" packets while the container is being restored without dropping them. As soon as the initial TCP SYN is dropped, the client will wait 1 second for retransmitting and make everything quite slow. I was unable to find a solution for this as of now so instead the userspace proxy is still required.

ctrox commented 10 months ago

this was merged but github did not detect it because I did not create a merge commit :shrug: