[Release-1.27] - Not able to connect to kubelet when using bind-address flag

[brandond]

Backport fix for Not able to connect to kubelet when using bind-address flag

• 10444

[aganesh-suse]

Validated on release-1.27 branch with commit 5b9a3fced6f8b98fa2fd469c0c23685498208f40

Environment Details

Infrastructure

• [x] Cloud
• [ ] Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
bind-address: 172.31.23.250
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

Testing Steps

1. Copy config.yaml

   $ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

   curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='5b9a3fced6f8b98fa2fd469c0c23685498208f40' sh -s - server

3. Verify Cluster Status:

   kubectl get nodes -o wide
   kubectl get pods -A

4. Verify we can perform 'kubectl logs' and 'kubectl exec' actions.

Replication Results:

• k3s version used for replication:

  $ k3s -v 
  k3s version v1.27.15+k3s1 (3ced503a)
  go version go1.21.11

$ sudo ss -lpn 'sport = :10250' 
Netid State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
tcp   LISTEN 0      4096   172.31.23.250:10250      0.0.0.0:*    users:(("k3s-server",pid=2840,fd=175))

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs pod/metrics-server-c44988498-k7tbh -n kube-system | grep 'v1beta1.metrics.k8s.io failed' 
Error from server: Get "https://172.31.23.250:10250/containerLogs/kube-system/metrics-server-c44988498-k7tbh/metrics-server": proxy error from 172.31.23.250:6443 while dialing 172.31.23.250:10250, code 502: 502 Bad Gateway

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml exec --namespace=kube-system local-path-provisioner-5c6c7f7cd-x7w2x -- sh -c 'ls' 
Error from server: error dialing backend: proxy error from 172.31.23.250:6443 while dialing 172.31.23.250:10250, code 502: 502 Bad Gateway

Validation Results:

• k3s version used for validation:

  $ k3s -v 
  k3s version v1.27.16-rc1+k3s1 (5b9a3fce)
  go version go1.22.5

$ sudo ss -lpn 'sport = :10250' 
Netid State  Recv-Q Send-Q Local Address:Port  Peer Address:PortProcess
tcp   LISTEN 0      4096   172.31.23.250:10250      0.0.0.0:*    users:(("k3s-server",pid=9199,fd=175))

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml logs pod/metrics-server-c44988498-5qd6k -n kube-system | grep 'v1beta1.metrics.k8s.io failed'

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml exec --namespace=kube-system local-path-provisioner-6fbb7c76c5-gghvs -- sh -c 'ls' 
bin
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var