k8s service: connection hangs after booting

I have a single node k8s-cluster and a teleport agent running outside from k8s but on the same node. When the node is started, k8s and the teleport agent are started at the same time. K8s takes longer to start than the agent. The agent uses a kubeconfig to connect to the cluster.

Expected behaviour:

As soon as k8s is started, requests can also be forwarded successfully to the kube-apiserver (default behaviour in teleport < 14)

Current behaviour:

When k8s is ready, it takes about 5 minutes until the agent can forward the requests to the kube-apiserver.

Bug details:

Teleport version (agent + server): 14.3.20
Recreation steps
1. start the teleport agent on your system (with configured kubernetes service)
2. start your k8s cluster
3. when the cluster is ready, it should take 5 min until the agent could forward the requests

Agent config:

kubernetes_service:
  enabled: true
  kubeconfig_file: "/secrets/kubernetes/admin/config"
  labels:
    usage: iiot
    env: dev

Debug logs (Agent logs)

 2024-06-11T12:23:38Z INFO             Starting Teleport v14.3.18 with a config file located at "/teleport-agent/config/config.yaml" common/teleport.go:632
 2024-06-11T12:23:39Z INFO [PROC:1]    Service diag is creating new listener on localhost:7755. pid:1.1 service/signals.go:242
 2024-06-11T12:23:39Z INFO [DIAG:1]    Starting diagnostic service on localhost:7755. pid:1.1 service/service.go:3201
 2024-06-11T12:23:39Z INFO [PROC:1]    Connecting to the cluster my.teleport.com with TLS client certificate. pid:1.1 service/connect.go:205
 2024-06-11T12:23:39Z INFO [PROC:1]    Connecting to the cluster my.teleport.com with TLS client certificate. pid:1.1 service/connect.go:205
 2024-06-11T12:23:39Z INFO [PROC:1]    Connecting to the cluster my.teleport.com with TLS client certificate. pid:1.1 service/connect.go:205
 2024-06-11T12:23:40Z INFO [PROC:1]    Instance: features loaded from auth server: Kubernetes:true App:true DB:true Desktop:true Assist:true DeviceTrust:<> AccessRequests:<> AccessList:<> AccessMonitoring:<> Policy:<>  pid:1.1 service/connect.go:1042
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  starting upload completer service pid:1.1 service/service.go:2918
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  Creating directory /teleport-agent/data/log. pid:1.1 service/service.go:2934
 2024-06-11T12:23:40Z INFO [INSTANCE:] Successfully registered instance client. pid:1.1 service/service.go:2513
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  Creating directory /teleport-agent/data/log/upload. pid:1.1 service/service.go:2934
 2024-06-11T12:23:40Z INFO [PROC:1]    Reusing Instance client for App. additionalSystemRoles=[App Kube] pid:1.1 service/connect.go:1013
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  Creating directory /teleport-agent/data/log/upload/streaming. pid:1.1 service/service.go:2934
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  Creating directory /teleport-agent/data/log/upload/streaming/default. pid:1.1 service/service.go:2934
 2024-06-11T12:23:40Z INFO [PROC:1]    Reusing Instance client for Kube. additionalSystemRoles=[App Kube] pid:1.1 service/connect.go:1013
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  Creating directory /teleport-agent/data/log. pid:1.1 service/service.go:2934
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  Creating directory /teleport-agent/data/log/upload. pid:1.1 service/service.go:2934
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  Creating directory /teleport-agent/data/log/upload/corrupted. pid:1.1 service/service.go:2934
 2024-06-11T12:23:40Z INFO [UPLOAD:1]  Creating directory /teleport-agent/data/log/upload/corrupted/default. pid:1.1 service/service.go:2934
 2024-06-11T12:23:40Z INFO [UPLOAD:1:] upload completer will run every 5m0s events/complete.go:159
 2024-06-11T12:23:40Z INFO [UPLOAD]    uploader will scan /teleport-agent/data/log/upload/streaming/default every 5s filesessions/fileasync.go:192
 2024-06-11T12:23:41Z INFO [APP:SERVI] Cache "apps" first init succeeded. cache/cache.go:997
 2024-06-11T12:23:41Z INFO [KUBERNETE] Cache "kube" first init succeeded. cache/cache.go:997
 2024-06-11T12:23:41Z INFO [KUBERNETE] Started reverse tunnel client. pid:1.1 service/kubernetes.go:148
 2024-06-11T12:23:41Z WARN [KUBERNETE] Failed to test the necessary Kubernetes permissions. The target Kubernetes cluster may be down or have misconfigured RBAC. This teleport instance will still handle Kubernetes requests towards this Kubernetes cluster. cluster:test-pqraus-vm error:[
 ERROR REPORT:
 Original Error: *url.Error Post &#34;https://test-pqraus-vm:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews&#34;: dial tcp 6443: connect: connection refused
 Stack Trace:
    github.com/gravitational/teleport/lib/kube/proxy/auth.go:247 github.com/gravitational/teleport/lib/kube/proxy.checkImpersonationPermissions
    github.com/gravitational/teleport/lib/kube/proxy/auth.go:163 github.com/gravitational/teleport/lib/kube/proxy.extractKubeCreds
    github.com/gravitational/teleport/lib/kube/proxy/auth.go:121 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).getKubeDetails
    github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:328 github.com/gravitational/teleport/lib/kube/proxy.NewForwarder
    github.com/gravitational/teleport/lib/kube/proxy/server.go:203 github.com/gravitational/teleport/lib/kube/proxy.NewTLSServer
    github.com/gravitational/teleport/lib/service/kubernetes.go:207 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initKubernetesService
    github.com/gravitational/teleport/lib/service/kubernetes.go:54 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initKubernetes.func1
    github.com/gravitational/teleport/lib/service/supervisor.go:583 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
    github.com/gravitational/teleport/lib/service/supervisor.go:310 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
    runtime/asm_amd64.s:1650 runtime.goexit
 User Message: failed to verify impersonation permissions for Kubernetes: Post &#34;https://test-pqraus-vm:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews&#34;: dial tcp 6443: connect: connection refused; this may be due to missing the SelfSubjectAccessReview permission on the ClusterRole used by the proxy; please make sure that proxy has all the necessary permissions: https://goteleport.com/teleport/docs/kubernetes-ssh/#impersonation
    Post &#34;https://test-pqraus-vm:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews&#34;: dial tcp 6443: connect: connection refused] pid:1.1 proxy/auth.go:164
 2024-06-11T12:23:41Z WARN [KUBERNETE] Failed to create cluster schema. Possibly the cluster is offline. cluster:test-pqraus-vm error:[
 ERROR REPORT:
 Original Error: *url.Error Get &#34;https://test-pqraus-vm:6443/api&#34;: dial tcp 6443: connect: connection refused
 Stack Trace:
    github.com/gravitational/teleport/lib/kube/proxy/scheme.go:138 github.com/gravitational/teleport/lib/kube/proxy.newClusterSchemaBuilder
    github.com/gravitational/teleport/lib/kube/proxy/cluster_details.go:125 github.com/gravitational/teleport/lib/kube/proxy.newClusterDetails
    github.com/gravitational/teleport/lib/kube/proxy/auth.go:134 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).getKubeDetails
    github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:328 github.com/gravitational/teleport/lib/kube/proxy.NewForwarder
    github.com/gravitational/teleport/lib/kube/proxy/server.go:203 github.com/gravitational/teleport/lib/kube/proxy.NewTLSServer
    github.com/gravitational/teleport/lib/service/kubernetes.go:207 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initKubernetesService
    github.com/gravitational/teleport/lib/service/kubernetes.go:54 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initKubernetes.func1
    github.com/gravitational/teleport/lib/service/supervisor.go:583 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
    github.com/gravitational/teleport/lib/service/supervisor.go:310 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
    runtime/asm_amd64.s:1650 runtime.goexit
 User Message: Get &#34;https://test-pqraus-vm:6443/api&#34;: dial tcp 6443: connect: connection refused] pid:1.1 proxy/cluster_details.go:127
 2024-06-11T12:23:41Z INFO [KUBERNETE] Starting Kube service via proxy reverse tunnel. pid:1.1 service/kubernetes.go:252
 2024-06-11T12:23:42Z INFO [APP:SERVI] All applications successfully started. pid:1.1 service/service.go:5506
 2024-06-11T12:23:42Z INFO [PROC:1]    The new service has started successfully. Starting syncing rotation status with period 10m0s. pid:1.1 service/connect.go:642
 2024-06-11T12:23:58Z INFO [AUDIT]     app.session.chunk app_name:private-test-pqraus-vm app_public_addr:private-test-pqraus-vm.my.teleport.com app_uri:http://10.96.0.100:80 
 2024-06-11T12:26:29Z WARN [KUBERNETE] Unable to setup context. error:[
 ERROR REPORT:
 Original Error: *trace.ConnectionProblemError kubernetes cluster &#34;test-pqraus-vm&#34; is offline
 Stack Trace:
    github.com/gravitational/teleport/lib/kube/proxy/cluster_details.go:194 github.com/gravitational/teleport/lib/kube/proxy.(*kubeDetails).getClusterSupportedResources
    github.com/gravitational/teleport/lib/kube/proxy/url.go:222 github.com/gravitational/teleport/lib/kube/proxy.getResourceFromRequest
    github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:844 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).parseResourceFromRequest
    github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:777 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).setupContext
    github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:543 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).authenticate
    github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:568 github.com/gravitational/teleport/lib/kube/proxy.NewForwarder.(*Forwarder).withAuthStd.func6
    github.com/gravitational/teleport/lib/httplib/httplib.go:145 github.com/gravitational/teleport/lib/kube/proxy.NewForwarder.(*Forwarder).withAuthStd.MakeStdHandlerWithErrorWriter.func8
    net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
    github.com/julienschmidt/httprouter@v1.3.0/router.go:460 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
    github.com/prometheus/client_golang@v1.19.0/prometheus/promhttp/instrument_server.go:296 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1
    net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
    github.com/prometheus/client_golang@v1.19.0/prometheus/promhttp/instrument_server.go:147 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
    net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
    github.com/prometheus/client_golang@v1.19.0/prometheus/promhttp/instrument_server.go:109 github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2
    net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
    github.com/prometheus/client_golang@v1.19.0/prometheus/promhttp/instrument_server.go:60 github.com/gravitational/teleport/lib/kube/proxy.instrumentHTTPHandlerWithPrometheus.InstrumentHandlerInFlight.func1
    net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:225 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:83 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1
    net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
    github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:400 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).ServeHTTP
    github.com/gravitational/teleport/lib/auth/middleware.go:711 github.com/gravitational/teleport/lib/auth.(*Middleware).ServeHTTP
    github.com/gravitational/oxy@v0.0.0-20221029012416-9fbf4c444680/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP
    github.com/gravitational/oxy@v0.0.0-20221029012416-9fbf4c444680/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP
    github.com/gravitational/teleport/lib/httplib/httplib.go:102 github.com/gravitational/teleport/lib/httplib.MakeTracingHandler.func1
    net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:225 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.49.0/handler.go:83 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1
    net/http/server.go:2136 net/http.HandlerFunc.ServeHTTP
    net/http/server.go:2938 net/http.serverHandler.ServeHTTP
    net/http/server.go:3546 net/http.initALPNRequest.ServeHTTP
    golang.org/x/net@v0.24.0/http2/server.go:2369 golang.org/x/net/http2.(*serverConn).runHandler
 User Message: kubernetes cluster &#34;test-pqraus-vm&#34; is offline] pid:1.1 proxy/forwarder.go:545
 2024-06-11T12:29:39Z INFO [KUBERNETE] Round trip: GET https://test-pqraus-vm:6443/api/v1/nodes?limit=500, code: 200, duration: 3.637756ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport.cluster.local pid:1.1 reverseproxy/reverse_proxy.go:221
 2024-06-11T12:29:45Z INFO [KUBERNETE] Round trip: GET https://test-pqraus-vm:6443/api/v1/namespaces/default/pods?limit=500, code: 200, duration: 2.567581ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport.cluster.local pid:1.1 reverseproxy/reverse_proxy.go:221

the request at 12:26:29 can't be handled
the last 2 requests can be handled
the kube-apiserver was ready at ~12:25

gravitational / teleport