search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
27.64k stars 2.32k forks source link

[Release-1.31] - Support for NodeExternalDNS #10854

Open brandond opened 1 week ago

brandond commented 1 week ago

Backport fix for Support for NodeExternalDNS

VestigeJ commented 2 days ago

New Feature

Environment Details

Validated using VERSION=v1.31.1-rc1+k3s1

Infrastructure

Node(s) CPU architecture, OS, and version:

Linux 6.4.0-150600.23.17-default x86_64 GNU/Linux PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"

Cluster Configuration:

NAME               STATUS   ROLES                       AGE   VERSION
ip-1-1-1-14        Ready    control-plane,etcd,master   20m   v1.31.1-rc1+k3s1

Config.yaml:

node-external-ip: 1.1.1.14
node-external-dns:
- cloudflare.com
- arc.k3s.com
token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
cluster-init: true
embedded-registry: true

Validation

``` $ curl https://get.k3s.io --output install-"k3s".sh $ sudo chmod +x install-"k3s".sh $ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd $ sudo modprobe ip_vs_rr $ sudo modprobe ip_vs_wrr $ sudo modprobe ip_vs_sh $ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf $ sudo cp 90-kubelet.conf /etc/sysctl.d/ $ sudo systemctl restart systemd-sysctl $ VERSION=v1.31.1-rc1+k3s1 $ sudo INSTALL_K3S_VERSION=$VERSION INSTALL_K3S_EXEC=server ./install-k3s.sh $ set_kubefig $ kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml $ k exec -it dnsutils -- nslookup cloudflare.com $ k exec -it dnsutils -- nslookup -debug cloudflare.com $ kgn -o yaml $ k exec -it dnsutils -- nslookup -debug arc.k3s.com $ k exec -it dnsutils -- nslookup -debug google.com ``` **Results:** $ kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml ``` pod/dnsutils created ``` $ kgp dnsutils ``` NAME READY STATUS RESTARTS AGE dnsutils 1/1 Running 0 6s ``` $ k exec -it dnsutils -- nslookup cloudflare.com ``` Server: 10.43.0.10 Address: 10.43.0.10#53 Non-authoritative answer: Name: cloudflare.com Address: 104.16.133.229 Name: cloudflare.com Address: 104.16.132.229 Name: cloudflare.com Address: 2611111:1111:::: Name: cloudflare.com Address: 2611:1111:::: ``` $ k exec -it dnsutils -- nslookup -debug cloudflare.com ``` Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: cloudflare.com.default.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1726525239 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find cloudflare.com.default.svc.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: cloudflare.com.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1726525239 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find cloudflare.com.svc.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: cloudflare.com.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1726525239 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find cloudflare.com.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: cloudflare.com.us-east-2.compute.internal, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> us-east-2.compute.internal origin = ns0.us-east-2.compute.internal mail addr = hostmaster.amazon.com serial = 2012103100 refresh = 3600 retry = 3600 expire = 3600 minimum = 60 ttl = 30 ADDITIONAL RECORDS: ------------ ** server can't find cloudflare.com.us-east-2.compute.internal: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: cloudflare.com, type = A, class = IN ANSWERS: -> cloudflare.com internet address = 104.16.132.229 ttl = 30 -> cloudflare.com internet address = 104.16.133.229 ttl = 30 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Non-authoritative answer: Name: cloudflare.com Address: 104.16.132.229 Name: cloudflare.com Address: 104.16.133.229 ------------ QUESTIONS: cloudflare.com, type = AAAA, class = IN ANSWERS: -> cloudflare.com has AAAA address 2611:1111:::: ttl = 30 -> cloudflare.com has AAAA address 2611111:1111:::: ttl = 30 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Name: cloudflare.com Address: 2611:1111:::: Name: cloudflare.com Address: 2611111:1111:::: ``` $ kgn -o yaml ``` apiVersion: v1 items: - apiVersion: v1 kind: Node metadata: annotations: alpha.kubernetes.io/provided-node-ip: 1.1.1.14,110:1111:1131:1101:119b:311:1191:1117 etcd.k3s.cattle.io/local-snapshots-timestamp: "2024-09-16T22:20:09Z" etcd.k3s.cattle.io/node-address: 1.1.1.14 etcd.k3s.cattle.io/node-name: ip-ip-8ead0505 flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"ea:3b:e6:61:1a:94"}' flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: "true" flannel.alpha.coreos.com/public-ip: 172.31.11.144 k3s.io/external-ip: 1.2.3.4 k3s.io/hostname: ip-ip k3s.io/internal-ip: 1.1.1.14,110:1111:1131:1101:119b:311:1191:1117 k3s.io/node-args: '["server","--node-external-ip","1.2.3.4","--node-external-dns","cloudflare.com","--node-external-dns","arc.k3s.com","--token","********","--write-kubeconfig-mode","644","--debug","true","--cluster-init","true","--embedded-registry","true"]' k3s.io/node-config-hash: 61111117Y565JXGI11111111111111111CWHBQ==== k3s.io/node-env: '{}' node.alpha.kubernetes.io/ttl: "0" volumes.kubernetes.io/controller-managed-attach-detach: "true" ``` Different example of a known bad dns lookup that continues hunting out further ``` ** server can't find arc.k3s.com.svc.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: arc.k3s.com.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1726525239 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find arc.k3s.com.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: arc.k3s.com.us-east-2.compute.internal, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> us-east-2.compute.internal origin = ns0.us-east-2.compute.internal mail addr = hostmaster.amazon.com serial = 2012103100 refresh = 3600 retry = 3600 expire = 3600 minimum = 60 ttl = 30 ADDITIONAL RECORDS: ------------ ** server can't find arc.k3s.com.us-east-2.compute.internal: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: arc.k3s.com, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> k3s.com origin = daisy.ns.cloudflare.com mail addr = dns.cloudflare.com serial = 2351630111 refresh = 10000 retry = 2400 expire = 604800 minimum = 1800 ttl = 30 ADDITIONAL RECORDS: ------------ Non-authoritative answer: ------------ QUESTIONS: arc.k3s.com, type = AAAA, class = IN ANSWERS: AUTHORITY RECORDS: -> k3s.com origin = daisy.ns.cloudflare.com mail addr = dns.cloudflare.com serial = 2351630111 refresh = 10000 retry = 2400 expire = 604800 minimum = 1800 ttl = 30 ADDITIONAL RECORDS: ------------ *** Can't find arc.k3s.com: No answer ``` $ k exec -it dnsutils -- nslookup -debug google.com ``` Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: google.com.default.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1726525239 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find google.com.default.svc.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: google.com.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1726525239 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find google.com.svc.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: google.com.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1726525239 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find google.com.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: google.com.us-east-2.compute.internal, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> us-east-2.compute.internal origin = ns0.us-east-2.compute.internal mail addr = hostmaster.amazon.com serial = 2012103100 refresh = 3600 retry = 3600 expire = 3600 minimum = 60 ttl = 28 ADDITIONAL RECORDS: ------------ ** server can't find google.com.us-east-2.compute.internal: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: google.com, type = A, class = IN ANSWERS: -> google.com internet address = 172.217.1.110 ttl = 30 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Non-authoritative answer: Name: google.com Address: 172.217.1.110 ------------ QUESTIONS: google.com, type = AAAA, class = IN ANSWERS: -> google.com has AAAA address 2607:f8b0:4009:81a::200e ttl = 30 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Name: google.com Address: 2607:f8b0:4009:81a::200e command terminated with exit code 1 ```