Tune cilium

[jfroy]

Pretty much everything in https://docs.cilium.io/en/stable/operations/performance/tuning

• netkit device mode: https://docs.cilium.io/en/stable/operations/performance/tuning/#netkit-device-mode
• eBPF host-routing: https://docs.cilium.io/en/latest/operations/performance/tuning/#ebpf-host-routing
• BIG TCP for IPv4/IPv6: https://docs.cilium.io/en/stable/operations/performance/tuning/#ipv6-big-tcp, https://docs.cilium.io/en/stable/operations/performance/tuning/#ipv4-big-tcp
• Bandwidth Manager: https://docs.cilium.io/en/latest/operations/performance/tuning/#bandwidth-manager
• BBR congestion control: https://docs.cilium.io/en/latest/operations/performance/tuning/#bbr-congestion-control-for-pods
• XDP Acceleration: https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/#xdp-acceleration
• terminatePodConnections: https://github.com/cilium/cilium/pull/33459, https://docs.cilium.io/en/latest/network/kubernetes/kubeproxy-free/#limitations

[JJGadgets]

Adding some of my personal experience, may be right or outdated or a skill issue on my part.

If you're using a simple L2 network setup (eg all networking configured on Talos config), then eBPF host routing is fine, but beware that if you do more complicated stuff on the host network layer, like receiving L3 routes via routing protocols like BGP/OSPF/OpenFabric etc, Cilium will not follow those properly with eBPF Masquerade enabled (it's flaky).

Also XDP acceleration may cause interfaces to not be detected and used properly if the XDP stuff isn't established as Cilium is starting up (eg having supported and unsupported interfaces in the mix and not specifying devices in Cilium config, or just the hardware + software stack used doesn't properly support XDP on all sides (maybe kernel config or whatnot)).

[jfroy]

Applied most of the changes except netkit which does not seem to work. Not root caused, but ARP seems to get blocked.

[jfroy]

Closing. Will open a separate issue for netkit once it has matured.