search

mholt / caddy-l4

Layer 4 (TCP/UDP) app for Caddy
Apache License 2.0
1.01k stars 75 forks source link

bug: TLS handshake with kubernetes apiserver times out intermittently #273

Open tetra12 opened 4 days ago

tetra12 commented 4 days ago

Hi! First off, thanks for building and supporting caddy and caddy-L4 :smiley:
We have been using caddy in production for about 2y for now and are totally happy :smile: with it

Now I wanna use caddy as an edge load balancer before kube apiserver. I have a pretty standard deployment as below: edge.

I've setup my laptop as a kube apiclient. It works, but fails due to TLS timeout intermittently, like one first command fails and have 2-3 commands succeeded.

Here's what I have:

 ~/.kube  kubectl get pods -A                                                                                                                            ✔  21:05:43 

E1124 21:05:55.183074  132224 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://public-IP:25025/api?timeout=32s\": net/http: TLS handshake timeout"
E1124 21:06:05.202558  132224 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://public-IP:25025/api?timeout=32s\": net/http: TLS handshake timeout"
E1124 21:06:15.221692  132224 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://public-IP:25025/api?timeout=32s\": net/http: TLS handshake timeout"
error: the server doesn't have a resource type "po"

 ~/.kube  kubectl get pods -A                                                                                                                          1 ✘  21:06:15 
NAMESPACE          NAME                                       READY   STATUS    RESTARTS      AGE
kube-system        coredns-7c65d6cfc9-7dfvp                   1/1     Running   1 (20h ago)   5d6h
kube-system        coredns-7c65d6cfc9-dx2cz                   1/1     Running   1 (20h ago)   5d6h
...

I don't really know how to debug this.

My setup:

OS (server): Ubuntu 24.04 OS(VM): Ubuntu 24.04

Caddyfile:

    # kube apiserver
    :25025 {
      route {
        proxy 192.168.122.10:6443
      }
    }

Caddy config:

caddy build-info

dep     github.com/caddyserver/caddy/v2         v2.8.4  h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk= 
dep     github.com/mholt/caddy-l4               v0.0.0-20241102143510-d8ba3fbdf35c      h1:3z5GznqFlQFOiyWdeVC7yYu1hWSZ7UHdS2dRUbvNCZg=
dep     github.com/mholt/caddy-ratelimit        v0.0.0-20240828171918-12435ecef5db      h1:30N0UnATYd7E8iaWSSOTlsr2/rd8v+7w0X+2Jc8FDJk=
tetra12 commented 4 days ago

this results in helm/api failing:

Error: Kubernetes cluster unreachable: Get "https://public-IP:25025/version": net/http: TLS handshake timeout