search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
28.08k stars 2.35k forks source link

Expose kube-controller-manager and kube-scheduler to external IP #11330

Closed fmunteanu closed 4 hours ago

fmunteanu commented 4 hours ago

Environmental Info: K3s Version: 

# k3s -v
k3s version v1.30.6+k3s1 (1829eaae)
go version go1.22.8

Node(s) CPU architecture, OS, and Version: 

# uname -a
Linux apollo 6.8.0-1014-raspi #16-Ubuntu SMP PREEMPT_DYNAMIC Tue Oct 15 20:54:23 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration: 3 servers, 5 agents

Describe the bug: My goal is to bind the kube-controller-manager and kube-scheduler ports to an external IP, so services like VictoriaMetrics can properly connect to them. Right now, only kube-etcd port 2379 is exposed to 192.168.4.2 external IP:

# ss -tulp | grep -E '2379|10257|10259'
tcp   LISTEN 0      4096          127.0.0.1:2379             0.0.0.0:*    users:(("k3s-server",pid=5290,fd=13))
tcp   LISTEN 0      4096          127.0.0.1:10257            0.0.0.0:*    users:(("k3s-server",pid=5290,fd=182))
tcp   LISTEN 0      4096          127.0.0.1:10259            0.0.0.0:*    users:(("k3s-server",pid=5290,fd=210))
tcp   LISTEN 0      4096        192.168.4.2:2379             0.0.0.0:*    users:(("k3s-server",pid=5290,fd=14))

Is there a K3s server configuration setting allowing me to the same for 10257 and 10259 ports?

Current server configuration:

# cat /etc/rancher/k3s/config.yaml
bind-address: 192.168.4.2
cluster-dns: 10.43.0.10
cluster-domain: cluster.local
disable:
  - local-storage
  - servicelb
  - traefik
disable-cloud-controller: true
disable-kube-proxy: true
disable-network-policy: true
embedded-registry: true
etcd-expose-metrics: true
flannel-backend: none
node-taint:
  - node.cilium.io/agent-not-ready:NoExecute
  - node-role.kubernetes.io/control-plane:NoSchedule
server: https://192.168.4.10:6443
tls-san:
  - 192.168.4.10
token: redacted
fmunteanu commented 4 hours ago

After looking closer at the server documentation, adding these values to /etc/rancher/k3s/config.yaml will expose correctly the services:

kube-controller-manager-arg: bind-address=192.168.4.2
kube-scheduler-arg: bind-address=192.168.4.2

Result:

# ss -tulpn | grep -E '2379|10257|10259'
tcp   LISTEN 0      4096          127.0.0.1:2379       0.0.0.0:*    users:(("k3s-server",pid=5420,fd=15))
tcp   LISTEN 0      4096        192.168.4.2:10259      0.0.0.0:*    users:(("k3s-server",pid=5420,fd=220))
tcp   LISTEN 0      4096        192.168.4.2:10257      0.0.0.0:*    users:(("k3s-server",pid=5420,fd=217))
tcp   LISTEN 0      4096        192.168.4.2:2379       0.0.0.0:*    users:(("k3s-server",pid=5420,fd=13))