search

kubesail / kubesail-agent

Allows a cluster or namespace to be managed by KubeSail.com
MIT License
30 stars 9 forks source link

Gateway can cause Linux Kernel "TCP: Out of Memory" issue #26

Open erulabs opened 2 years ago

erulabs commented 2 years ago

KubeSail Gateway can occasionally buffer too much memory in TCP sockets to packet-lossy or mis-behaving connected agents.

A real fix would be to update gateway handleSocket() to keep a highwaterMark for buffered packets (or rather, to track it), and pause the incoming socket when the destination socket is queuing.

Until then, a temporary fix is to dramatically increase the TCP buffer size in the linux kernel of Gateway servers:

net.core.netdev_max_backlog=30000
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 87380 67108864

Note that kubesail-agent users don't need to worry about any of this!