search

rancher / k3os

Purpose-built OS for Kubernetes, fully managed by Kubernetes.
https://k3os.io
Apache License 2.0
3.5k stars 403 forks source link

Worker node unable to establish connections to the internet #726

Open sj14 opened 3 years ago

sj14 commented 3 years ago

Version (k3OS / kernel)

k3os version v0.20.7-k3s1r0 5.4.0-73-generic #82 SMP Thu Jun 3 01:19:50 UTC 2021

Architecture

aarch64

Describe the bug

Issues with establishing outgoing connections on the worker node. For example, when trying to start a pod, pulling the image fails:

Warning  Failed     4s    kubelet            Error: ErrImagePull
  Warning  Failed     4s    kubelet            Failed to pull image "alpine": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/alpine:latest": failed to resolve reference "docker.io/library/alpine:latest": failed to d
o request: Head "https://registry-1.docker.io/v2/library/alpine/manifests/latest": dial tcp 54.85.56.253:443: i/o timeout

Shell on the main machine

k3os-1 [~]$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N KUBE-EXTERNAL-SERVICES
-N KUBE-FIREWALL
-N KUBE-FORWARD
-N KUBE-KUBELET-CANARY
-N KUBE-PROXY-CANARY
-N KUBE-ROUTER-FORWARD
-N KUBE-ROUTER-INPUT
-N KUBE-ROUTER-OUTPUT
-N KUBE-SERVICES
-A INPUT -m comment --comment "kube-router netpol - 4IA2OSFRMVNDXBVV" -j KUBE-ROUTER-INPUT
-A INPUT -j KUBE-FIREWALL
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -m comment --comment "kube-router netpol - TEMCG2JMHZYE7H7T" -j KUBE-ROUTER-FORWARD
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A OUTPUT -m comment --comment "kube-router netpol - VEAAIY32XVBHCSCY" -j KUBE-ROUTER-OUTPUT
-A OUTPUT -j KUBE-FIREWALL
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-ROUTER-INPUT -d 10.43.0.0/16 -m comment --comment "allow traffic to cluster IP - M66LPN4N3KB5HTJR" -j RETURN
-A KUBE-ROUTER-INPUT -p tcp -m comment --comment "allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M" -m addrtype --dst-type LOCAL -m multiport --dports 30000:32767 -j RETURN
-A KUBE-ROUTER-INPUT -p udp -m comment --comment "allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ" -m addrtype --dst-type LOCAL -m multiport --dports 30000:32767 -j RETURN
k3os-1 [~]$ ifconfig
cni0      Link encap:Ethernet  HWaddr E6:01:F3:DD:DF:1D
          inet addr:10.42.0.1  Bcast:10.42.0.255  Mask:255.255.255.0
          inet6 addr: fe80::e401:f3ff:fedd:df1d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:157984 errors:0 dropped:0 overruns:0 frame:0
          TX packets:155284 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:46494451 (44.3 MiB)  TX bytes:86318507 (82.3 MiB)

eth0      Link encap:Ethernet  HWaddr 02:00:17:02:9B:C6
          inet addr:10.0.0.236  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::17ff:fe02:9bc6/64 Scope:Link
          inet6 addr: 2603:c020:8001:60ff:91e8:47f6:6c5e:c36f/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:127162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:81289 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:76288552 (72.7 MiB)  TX bytes:39300298 (37.4 MiB)

flannel.1 Link encap:Ethernet  HWaddr 46:E3:D5:A1:B1:2E
          inet addr:10.42.0.0  Bcast:0.0.0.0  Mask:255.255.255.255
          inet6 addr: fe80::44e3:d5ff:fea1:b12e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:5312 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4157 errors:0 dropped:7 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:559168 (546.0 KiB)  TX bytes:7240707 (6.9 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:403952 errors:0 dropped:0 overruns:0 frame:0
          TX packets:403952 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:160486679 (153.0 MiB)  TX bytes:160486679 (153.0 MiB)

veth45f917c6 Link encap:Ethernet  HWaddr 8E:77:B5:99:0F:43
          inet6 addr: fe80::8c77:b5ff:fe99:f43/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:17 errors:0 dropped:0 overruns:0 frame:0
          TX packets:195 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1258 (1.2 KiB)  TX bytes:11374 (11.1 KiB)

veth4bc0501e Link encap:Ethernet  HWaddr EE:05:94:95:2E:CA
          inet6 addr: fe80::ec05:94ff:fe95:2eca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:27482 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30020 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:11805648 (11.2 MiB)  TX bytes:5349978 (5.1 MiB)

veth6097774f Link encap:Ethernet  HWaddr 5E:CC:9C:85:39:E8
          inet6 addr: fe80::5ccc:9cff:fe85:39e8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:21571 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22195 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1962339 (1.8 MiB)  TX bytes:7635885 (7.2 MiB)

veth72d69691 Link encap:Ethernet  HWaddr 1A:EB:DB:09:57:48
          inet6 addr: fe80::18eb:dbff:fe09:5748/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:320 errors:0 dropped:0 overruns:0 frame:0
          TX packets:619 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:25229 (24.6 KiB)  TX bytes:44257 (43.2 KiB)

veth87888582 Link encap:Ethernet  HWaddr 22:4A:29:81:2F:F0
          inet6 addr: fe80::204a:29ff:fe81:2ff0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:21494 errors:0 dropped:0 overruns:0 frame:0
          TX packets:21114 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5062711 (4.8 MiB)  TX bytes:8247863 (7.8 MiB)

vethb990a52d Link encap:Ethernet  HWaddr FA:1C:05:CA:2C:61
          inet6 addr: fe80::f81c:5ff:feca:2c61/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:5057 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4951 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:854450 (834.4 KiB)  TX bytes:1837873 (1.7 MiB)

vethc2134630 Link encap:Ethernet  HWaddr BE:33:E4:7A:3B:1E
          inet6 addr: fe80::bc33:e4ff:fe7a:3b1e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:51485 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50357 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4849546 (4.6 MiB)  TX bytes:8264774 (7.8 MiB)

vethdee2816a Link encap:Ethernet  HWaddr 66:10:CA:8E:65:16
          inet6 addr: fe80::6410:caff:fe8e:6516/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:27721 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23027 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:24379846 (23.2 MiB)  TX bytes:15604591 (14.8 MiB)

vethfbb585df Link encap:Ethernet  HWaddr B2:AD:9E:21:CC:6D
          inet6 addr: fe80::b0ad:9eff:fe21:cc6d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:7096 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8697 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2127456 (2.0 MiB)  TX bytes:1204999 (1.1 MiB)

vethff268cbc Link encap:Ethernet  HWaddr D6:3A:46:67:0A:24
          inet6 addr: fe80::d43a:46ff:fe67:a24/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:13049 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13389 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3285852 (3.1 MiB)  TX bytes:3307056 (3.1 MiB)
k3os-1 [~]$ ip route list
default via 10.0.0.1 dev eth0
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.236
10.0.0.1 dev eth0 scope link
10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink
10.42.2.0/24 via 10.42.2.0 dev flannel.1 onlink
127.0.0.0/8 dev lo scope host
169.254.169.254 via 10.0.0.1 dev eth0
k3os-1 [~]$ sudo traceroute google.com
traceroute to google.com (142.250.186.174), 30 hops max, 46 byte packets
 1  140.91.198.103 (140.91.198.103)  0.096 ms  140.91.198.106 (140.91.198.106)  0.091 ms  140.91.198.111 (140.91.198.111)  0.162 ms
 2  185.1.102.135 (185.1.102.135)  0.719 ms  ipv4.de-cix.fra.de.as31898.oracle.com (80.81.196.168)  0.319 ms  13.348 ms
 3  185.1.102.59 (185.1.102.59)  0.990 ms  0.615 ms  ipv4.de-cix.fra.de.as15169.google.com (80.81.192.108)  0.454 ms
 4  108.170.251.129 (108.170.251.129)  0.644 ms  108.170.252.1 (108.170.252.1)  3.295 ms  1.701 ms
 5  142.250.214.203 (142.250.214.203)  0.656 ms  0.676 ms  142.250.214.201 (142.250.214.201)  0.612 ms
 6  fra24s08-in-f14.1e100.net (142.250.186.174)  0.646 ms  0.641 ms  0.612 ms
k3os-1 [~]$ ping google.com
PING google.com (142.250.186.174): 56 data bytes
64 bytes from 142.250.186.174: seq=0 ttl=42 time=0.696 ms
64 bytes from 142.250.186.174: seq=1 ttl=42 time=0.788 ms
64 bytes from 142.250.186.174: seq=2 ttl=42 time=0.763 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.696/0.749/0.788 ms

Shell on the worker machine

k3os-3 [~]$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N KUBE-EXTERNAL-SERVICES
-N KUBE-FIREWALL
-N KUBE-FORWARD
-N KUBE-KUBELET-CANARY
-N KUBE-PROXY-CANARY
-N KUBE-ROUTER-FORWARD
-N KUBE-ROUTER-INPUT
-N KUBE-ROUTER-OUTPUT
-N KUBE-SERVICES
-A INPUT -m comment --comment "kube-router netpol - 4IA2OSFRMVNDXBVV" -j KUBE-ROUTER-INPUT
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kube-router netpol - TEMCG2JMHZYE7H7T" -j KUBE-ROUTER-FORWARD
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A OUTPUT -m comment --comment "kube-router netpol - VEAAIY32XVBHCSCY" -j KUBE-ROUTER-OUTPUT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-ROUTER-INPUT -d 10.43.0.0/16 -m comment --comment "allow traffic to cluster IP - M66LPN4N3KB5HTJR" -j RETURN
-A KUBE-ROUTER-INPUT -p tcp -m comment --comment "allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M" -m addrtype --dst-type LOCAL -m multiport --dports 30000:32767 -j RETURN
-A KUBE-ROUTER-INPUT -p udp -m comment --comment "allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ" -m addrtype --dst-type LOCAL -m multiport --dports 30000:32767 -j RETURN
k3os-3 [~]$ ifconfig
cni0      Link encap:Ethernet  HWaddr EE:AD:96:62:61:A3
          inet addr:10.42.2.1  Bcast:10.42.2.255  Mask:255.255.255.0
          inet6 addr: fe80::ecad:96ff:fe62:61a3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:7916 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7798 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7319918 (6.9 MiB)  TX bytes:7426342 (7.0 MiB)

eth0      Link encap:Ethernet  HWaddr 02:00:17:02:90:DF
          inet addr:10.0.0.48  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::17ff:fe02:90df/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:33095 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15755 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:37844569 (36.0 MiB)  TX bytes:8658684 (8.2 MiB)

flannel.1 Link encap:Ethernet  HWaddr DA:56:C8:8D:B6:46
          inet addr:10.42.2.0  Bcast:0.0.0.0  Mask:255.255.255.255
          inet6 addr: fe80::d856:c8ff:fe8d:b646/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:3638 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4268 errors:0 dropped:7 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6971760 (6.6 MiB)  TX bytes:356069 (347.7 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:6347 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6347 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2555188 (2.4 MiB)  TX bytes:2555188 (2.4 MiB)

veth4c986d72 Link encap:Ethernet  HWaddr CA:B3:A7:B3:A7:DA
          inet6 addr: fe80::c8b3:a7ff:feb3:a7da/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1048 (1.0 KiB)  TX bytes:3024 (2.9 KiB)

veth5596d2bd Link encap:Ethernet  HWaddr 72:BD:46:A8:39:47
          inet6 addr: fe80::70bd:46ff:fea8:3947/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:158 errors:0 dropped:0 overruns:0 frame:0
          TX packets:142 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:14753 (14.4 KiB)  TX bytes:17211 (16.8 KiB)

veth598e4a70 Link encap:Ethernet  HWaddr 06:A9:29:0B:1A:BE
          inet6 addr: fe80::4a9:29ff:fe0b:1abe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:7657 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7713 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7407895 (7.0 MiB)  TX bytes:7412487 (7.0 MiB)

vethb8f2a59a Link encap:Ethernet  HWaddr DA:05:AD:1D:23:FA
          inet6 addr: fe80::d805:adff:fe1d:23fa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:87 errors:0 dropped:0 overruns:0 frame:0
          TX packets:104 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7046 (6.8 KiB)  TX bytes:9378 (9.1 KiB)
k3os-3 [~]$ ip route list
default via 10.0.0.1 dev eth0
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.48
10.0.0.1 dev eth0 scope link
10.42.0.0/24 via 10.42.0.0 dev flannel.1 onlink
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink
10.42.2.0/24 dev cni0 proto kernel scope link src 10.42.2.1
127.0.0.0/8 dev lo scope host
169.254.169.254 via 10.0.0.1 dev eth0
k3os-3 [~]$ sudo traceroute google.com
traceroute to google.com (142.250.186.142), 30 hops max, 46 byte packets
 1  140.91.198.107 (140.91.198.107)  0.163 ms  140.91.198.158 (140.91.198.158)  0.144 ms  140.91.198.108 (140.91.198.108)  0.140 ms
 2  185.1.102.135 (185.1.102.135)  0.552 ms  ipv4.de-cix.fra.de.as31898.oracle.com (80.81.196.168)  0.332 ms  0.397 ms
 3  185.1.102.59 (185.1.102.59)  0.551 ms  0.584 ms  ipv4.de-cix.fra.de.as15169.google.com (80.81.192.108)  1.288 ms
 4  108.170.252.1 (108.170.252.1)  1.768 ms  1.858 ms  108.170.251.129 (108.170.251.129)  0.676 ms
 5  142.250.214.197 (142.250.214.197)  0.606 ms  0.674 ms  142.250.214.195 (142.250.214.195)  0.572 ms
 6  fra24s07-in-f14.1e100.net (142.250.186.142)  0.643 ms  0.607 ms  0.596 ms
k3os-3 [~]$ ping google.com
PING google.com (142.250.186.142): 56 data bytes
64 bytes from 142.250.186.142: seq=0 ttl=42 time=0.632 ms
64 bytes from 142.250.186.142: seq=1 ttl=42 time=0.759 ms
64 bytes from 142.250.186.142: seq=2 ttl=42 time=0.728 ms
64 bytes from 142.250.186.142: seq=3 ttl=42 time=0.758 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.632/0.719/0.759 ms

Shell on svclb-traefik pod on the main node:

/ # ifconfig
eth0      Link encap:Ethernet  HWaddr A2:49:3E:F1:97:D8
          inet addr:10.42.0.47  Bcast:10.42.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a049:3eff:fef1:97d8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:642 errors:0 dropped:0 overruns:0 frame:0
          TX packets:343 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:47495 (46.3 KiB)  TX bytes:28467 (27.7 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
/ # ip route list
default via 10.42.0.1 dev eth0
10.42.0.0/24 dev eth0 scope link  src 10.42.0.47
10.42.0.0/16 via 10.42.0.1 dev eth0
/ # traceroute google.com
traceroute to google.com (172.217.16.142), 30 hops max, 46 byte packets
 1  10.42.0.1 (10.42.0.1)  0.003 ms  0.003 ms  0.001 ms
 2  140.91.198.99 (140.91.198.99)  0.102 ms  140.91.198.102 (140.91.198.102)  0.106 ms  0.101 ms
 3  ip4.gtt.net (154.14.43.66)  0.583 ms  0.590 ms  0.546 ms
 4  ae23-3201.cr2-fra6.ip4.gtt.net (154.14.43.65)  1.322 ms  1.095 ms  0.695 ms
 5  ae10.cr6-fra2.ip4.gtt.net (141.136.107.233)  0.704 ms  ae9.cr6-fra2.ip4.gtt.net (141.136.110.41)  1.294 ms  16.245 ms
 6  ip4.gtt.net (46.33.79.250)  0.666 ms  0.566 ms  0.550 ms
 7  *  *  *
 8  142.251.64.182 (142.251.64.182)  2.809 ms  142.251.64.184 (142.251.64.184)  0.911 ms  108.170.252.65 (108.170.252.65)  2.393 ms
 9  66.249.94.245 (66.249.94.245)  1.412 ms  1.491 ms  108.170.252.83 (108.170.252.83)  1.675 ms
10  fra15s46-in-f14.1e100.net (172.217.16.142)  0.855 ms  0.874 ms  0.891 ms
/ # ping google.com
PING google.com (142.250.186.46): 56 data bytes
64 bytes from 142.250.186.46: seq=0 ttl=121 time=0.770 ms
64 bytes from 142.250.186.46: seq=1 ttl=121 time=0.937 ms
64 bytes from 142.250.186.46: seq=2 ttl=121 time=0.852 ms
64 bytes from 142.250.186.46: seq=3 ttl=121 time=0.839 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.770/0.849/0.937 ms

Shell on svclb-traefik pod on the worker node:

/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 6A:82:6A:E8:E2:DA
          inet addr:10.42.2.2  Bcast:10.42.2.255  Mask:255.255.255.0
          inet6 addr: fe80::6882:6aff:fee8:e2da/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:101 errors:0 dropped:0 overruns:0 frame:0
          TX packets:87 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:9196 (8.9 KiB)  TX bytes:7046 (6.8 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
/ # ip route list
default via 10.42.2.1 dev eth0
10.42.0.0/16 via 10.42.2.1 dev eth0
10.42.2.0/24 dev eth0 scope link  src 10.42.2.2
/ # traceroute google.com
traceroute to google.com (172.217.16.142), 30 hops max, 46 byte packets
 1  10.42.2.1 (10.42.2.1)  0.004 ms  0.003 ms  0.001 ms
 2  *  *  *
 3  *  *  *
 4^C
/ # ping google.com
PING google.com (172.217.16.142): 56 data bytes
^C
--- google.com ping statistics ---
35 packets transmitted, 0 packets received, 100% packet loss

To Reproduce

Create a main node

hostname: k3os
k3os:
  dns_nameservers:
  - 8.8.8.8
  - 1.1.1.1
  k3s_args:
  - server
  - --node-ip=10.0.0.236
  - --advertise-address=10.0.0.236
  modules:
  - kvm
  - nvme
  ntp_servers:
  - 0.us.pool.ntp.org
  - 1.us.pool.ntp.org
  sysctls:
    kernel.kptr_restrict: "1"
    kernel.printk: 4 4 1 7
  token: TOKEN_VALUE
ssh_authorized_keys:
- github:sj14

Create a worker node

hostname: k3os-worker
k3os:
  dns_nameservers:
    - 8.8.8.8
    - 1.1.1.1
  k3s_args:
    - agent
    - --node-ip=10.0.0.48
  modules:
    - kvm
    - nvme
  ntp_servers:
    - 0.us.pool.ntp.org
    - 1.us.pool.ntp.org
  server_url: https:////10.0.0.236:6443
  sysctls:
    kernel.kptr_restrict: "1"
    kernel.printk: 4 4 1 7
  token: XXX::server:XXX
ssh_authorized_keys:
  - github:sj14

Try to establish an outside connection from the worker node.

Expected behavior

A successful connection to the internet from the pods on the worker node.

Actual behavior

The connections time out.

Additional context

Executing this line fixed the issue (https://github.com/k3s-io/k3s/issues/24#issuecomment-469759329):

iptables -I INPUT 1 -i cni0 -s 10.42.0.0/16 -j ACCEPT

It would be great when this rule would be installed and persisted automatically. When this is not possible or my setup is broken, please let me know. I'm still checking how I can persist this rule to survive a reboot.

sj14 commented 3 years ago

Joining the nodes as master nodes with server instead of agent args also seems to prevent from this issue.