CPLB hairpining fails on vSphere

juanluisvaladas commented 4 days ago

Problem statement

After some investigation it was found that the all hosts reply to ARP requests to the virtual IP because the host has that virtual ip in the interface dummyvip0.

This behavior can be avoided easily by setting the sysctl net.ipv4.conf.<interface>.arp_ignore = 1. However this "randomly" breaks hairpining with IPVS. On other circumstances this could be avoided entirely using iptables nat table with PREROUTING and POSTROUTING rules, however in the scenario of calico + kube-proxy I have not been able to do this.

I think we'll need to have the following behavior: 1- When CPLB runs in controller+worker, forbid virtualServers in keepalived 2- For load balancing, introduce a reverse proxy. This could be something external like haproxy, nginx or envoy or an embedded library.

Reverse proxy

Golang library:

1- inetaf tcpproxy: Very simple, well written and uses Apache 2 license (my favourite option) 2- Caddy: links kubernetes libraries and isn't even up to date. Most likely very painful in the future 3- Golang's httputil/reverseproxy.go). We need a TCP reverse proxy because mTLS. We simply can't use this. 4- Kube-proxy: Userspace proxy doesn't exist anymore

External process

1- Nginx: Runs on everything except RISC-V (I've seen a fork on github) alpine build is less than 1Mb uncompressed and less than 500k. It has a few dependencies which aren't statically linked but I doubt it'd get much larger anyway:

/ # ldd  /usr/sbin/nginx
    /lib/ld-musl-x86_64.so.1 (0x7f3ab6e70000)
    libpcre.so.1 => /usr/lib/libpcre.so.1 (0x7f3ab6cfb000)
    libssl.so.3 => /lib/libssl.so.3 (0x7f3ab6c75000)
    libcrypto.so.3 => /lib/libcrypto.so.3 (0x7f3ab6858000)
    libz.so.1 => /lib/libz.so.1 (0x7f3ab683e000)
    libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f3ab6e70000)

2- Apache: similar numbers, haven't found any information about RISC-V. 3- Haproxy: Doesn't say anything about RISC-V support but I've seen compilations for RISC-V, I'm not sure if they are modified. Haproxy build for alpine is 3.2Mb uncompressed and 1.4Mb compressed. It links dynamically a few libraries:

/ # ldd /usr/sbin/haproxy
    /lib/ld-musl-x86_64.so.1 (0x7f8371f00000)
    libssl.so.3 => /lib/libssl.so.3 (0x7f83718da000)
    libcrypto.so.3 => /lib/libcrypto.so.3 (0x7f83714bd000)
    liblua-5.4.so.0 => /usr/lib/liblua-5.4.so.0 (0x7f8371487000)
    libz.so.1 => /lib/libz.so.1 (0x7f837146d000)
    libpcre2-8.so.0 => /usr/lib/libpcre2-8.so.0 (0x7f83713c6000)
    libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f8371f00000)

it could be statically linked and we can probably compile it in a way that doesn't require some of these...

4- Envoy: Envoy latest release nearly 70Mb for x86 architecture, gzip compressed around 24Mb and it also has limitations on the architectures it can run on.

jnummelin commented 3 days ago

My go-to choice would be https://github.com/inetaf/tcpproxy/ for couple reasons:

small in simple, but still suits our needs (AFAIK)
Pure Golang, no issues in future archs like RISCV etc.
Minimal deps (only one)

ncopa commented 1 day ago

Nginx: Runs on everything except RISC-V

I'm pretty sure nginx runs on RISC-V.

alpine-ci-rv-2:~# uname -a
Linux alpine-ci-rv-2 5.10.113+ #1 SMP PREEMPT Tue Apr 23 11:57:43 UTC 2024 riscv64 Linux
alpine-ci-rv-2:~# apk add nginx
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/riscv64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/riscv64/APKINDEX.tar.gz
(1/4) Installing pcre (8.45-r3)
(2/4) Installing nginx (1.26.1-r2)
Executing nginx-1.26.1-r2.pre-install
Executing nginx-1.26.1-r2.post-install
(3/4) Installing nginx-openrc (1.26.1-r2)
(4/4) Installing nginx-vim (1.26.1-r2)
Executing busybox-1.36.1-r26.trigger
OK: 354 MiB in 105 packages
alpine-ci-rv-2:~# nginx -version
nginx version: nginx/1.26.1

k0sproject / k0s