Can't reach internet from pod / container

apiening commented 2 years ago

Environmental Info: K3s Version:

k3s -v
k3s version v1.22.7+k3s1 (8432d7f2)
go version go1.16.10

Host OS Version:

cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

IP Forwarding:

# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

Node(s) CPU architecture, OS, and Version:

Linux ansible-awx 5.4.0-105-generic #119-Ubuntu SMP Mon Mar 7 18:49:24 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: Single node.

# k3s kubectl get nodes -o wide
NAME          STATUS   ROLES                  AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
ansible-awx   Ready    control-plane,master   5d10h   v1.22.7+k3s1   10.164.12.6   <none>        Ubuntu 20.04.4 LTS   5.4.0-105-generic   containerd://1.5.9-k3s1

Describe the bug: I cannot connect to the internet from within the pod / container:

# time curl https://www.google.de
curl: (7) Failed to connect to www.google.de port 443: Connection timed out

real    2m11.892s
user    0m0.005s
sys     0m0.005s

Steps To Reproduce: Install one node k3s cluster with curl -sfL https://get.k3s.io | sh on a ubuntu 20.04 VM. Setup a simple workload (in my case AWX - https://github.com/ansible/awx-operator#basic-install-on-existing-cluster). Enter a container and try to access the internet (for example with curl on a public address).

Expected behavior: Accessing the internet should be working the same way like it is from the host.

Actual behavior: No connectivity to the internet from the pod / container at all.

Additional context / logs:

# cat /etc/resolv.conf 
search awx.svc.cluster.local svc.cluster.local cluster.local mydomain.com
nameserver 10.43.0.10
options ndots:5

malyem commented 2 years ago

I have had similar issue. On my server tcpdump showed only SYN outgoing packages without SYN ACK on public interface, but not every time. The reason of problem was firewall which I can setup for the sever machine in providers firewall (not in local system). One of the rules was allowing outgoing traffic from ports 32678-65535 only. Most outgoing connections are using this port range, but I don't known why k3s doesn't.

apiening commented 2 years ago

I have had similar issue. On my server tcpdump showed only SYN outgoing packages without SYN ACK on public interface, but not every time. The reason of problem was firewall which I can setup for the sever machine in providers firewall (not in local system). One of the rules was allowing outgoing traffic from ports 32678-65535 only. Most outgoing connections are using this port range, but I don't known why k3s doesn't.

What you described turned out to be the reason for my issues as well! My provider (hetzner) had a rule for incoming TCP ports ranging from 32768 to 65535 with the flag ack in the default template, which was applied to my server. After changing the port range starting from 32768 to 0 the connection tests worked reliably. This perfectly explains, why some attempts did work before while others don't: If the randomly selected port was in the upper (not blocked) range, it worked. For the ports below 32768 it did not. This has never been an issue before for me before. It still seems strange to me why the VM running k3s, the host and all other VMs are using the upper half of the available ports and k3s seems to use the full range of possible ports.

However, thank you all for your help and support!

I'm really happy that this issue is solved.

manuelbuil commented 2 years ago

This is really interesting. The linux kernel by default uses the range 32768 to 60999 (checkcat /proc/sys/net/ipv4/ip_local_port_range) for client TCP connections. However, iptables, when using the flag --random or --random-fully replaces the source TCP port when doing the SNAT with whatever unassigned port, but it doesn't have to be in that range. Flannel uses that flag to do SNAT. I wonder how other CNI plugins do it.... but at least we should document this to avoid more users having this issue

manuelbuil commented 2 years ago

While testing calico, I can also see:

-A cali-POSTROUTING -o vxlan.calico -m comment --comment "cali:e9dnSgSVNmIcpVhP" -m addrtype ! --src-type LOCAL --limit-iface-out -m addrtype --src-type LOCAL -j MASQUERADE --random-fully
-A cali-nat-outgoing -m comment --comment "cali:flqWnvo8yq4ULQLa" -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst -j MASQUERADE --random-fully

And after testing 3 times: 16:53:57.224979 IP 10.0.10.40.20507 > 142.250.74.227.443: Flags [S], seq 3466148145, win 62377, options [mss 8911,sackOK,TS val 100234485 ecr 0,nop,wscale 7], length 0

i.e. port 20507, which is below 32768 but is unassigned ==> https://www.speedguide.net/port.php?port=20507

apiening commented 2 years ago

Thank you @manuelbuil for the explanation. The fact that ports above 32768 are the default port range for linux (and I guess other OS as well) also explains why I never ran into an issue before. One simple reason for doing --random-fully may be just the larger range that allows more ports to be opened. I don't use more than 32k ports in my cluster, but large ones may hit this limit.

This was really tough to debug for me, but at least I've learned something along the way. Thanks again.

manuelbuil commented 2 years ago

Thank you @manuelbuil for the explanation. The fact that ports above 32768 are the default port range for linux (and I guess other OS as well) also explains why I never ran into an issue before. One simple reason for doing --random-fully may be just the larger range that allows more ports to be opened. I don't use more than 32k ports in my cluster, but large ones may hit this limit.

This was really tough to debug for me, but at least I've learned something along the way. Thanks again.

I learnt something too, so thanks for reporting it ;)

wind57 commented 2 years ago

Someone should explain for mere mortals what we should do. :(

For example I am running testcontainers with rancher/k3s:v1.21.10-k3s1. I can docker exec into the container once it is started, but inside it ping fails.

On the other hand if I do kubectl run ... busybox (provided that I mount a directory where the .tar busybox is present and then issue ctr i import of that tar), ping works just fine over there. thank you.

wind57 commented 2 years ago

nvm... only ping is disabled, everything else works. silly me!

DavidPerezIngeniero commented 1 year ago

In my case, after rebooting my machine, pods have Internet again.

k3s v1.25.4+k3s1
Ubuntu 22.04 as host OS.
Docker version 20.10.22

malt3 commented 1 year ago

Thank you so much. I ran into the same issue.

Gabgobie commented 1 year ago

Thank you all so much for not leaving this thread to die before the issue was resolved. I ran into what I believe to be the same or a similar issue as now everything at least partially works.

My pods were unable to dial up anything outside the cluster entirely and eversince I flushed the iptables and set everything to accept at least pings work. I still can't use any dns except for the cluster dns which is kind of confusing and wget refuses to solve hostnames on its own but will be able to fetch the page if I provide the ip address myself after using nslookup inside the same pod but that is something for me to figure out tomorrow.

All in all massive thanks to you! This helped me a lot and I would have never thought that it might have been the port range.

raine commented 1 year ago

@apiening

What you described turned out to be the reason for my issues as well! My provider (hetzner) had a rule for incoming TCP ports ranging from 32768 to 65535 with the flag ack in the default template, which was applied to my server. After changing the port range starting from 32768 to 0 the connection tests worked reliably.

Could you explain where the rule is configured? I'm also on hetzner and believe I have the same issue.

apiening commented 1 year ago

@apiening

What you described turned out to be the reason for my issues as well! My provider (hetzner) had a rule for incoming TCP ports ranging from 32768 to 65535 with the flag ack in the default template, which was applied to my server. After changing the port range starting from 32768 to 0 the connection tests worked reliably.

Could you explain where the rule is configured? I'm also on hetzner and believe I have the same issue.

If you select the server in the management console for dedicated servers (https://robot.hetzner.com/server) you can select the tab "Firewall" to edit the rules.

raine commented 1 year ago

Thank you. However, I have a virtual server, so I have nothing there. It's an arm64 based cloud server that Hetzner recently introduced in Falkenstein. Very strange.

My symptom is that inside a container running with k3s, curl https://api.telegram.org/ takes 5 seconds to complete, but also very rarely it's instant.

root@app-649b9f7885-j5x4z:~# time curl https://api.telegram.org/
...

real    0m5.115s

Running the curl command directly in the node is invariably instant.

I think I will have to try to cilium workaround mentioned.

raine commented 1 year ago

I reinstalled k3s with flannel disabled, installed Cilium and the issue persists.

Does anyone happen to have any ideas? 🙏

raine commented 1 year ago

I have managed to figure out the reason but not the root cause.

Turns out for me it's unrelated to Flannel or Hetzner firewall settings and instead it appears to have something to do with k3s and the debian:buster docker image.

Running a pod with `debian:buster` (has delay)

$ kubectl run curl --rm --attach -it --image=debian:buster --restart=Never --command -- /bin/sh -c "apt-get update && apt-get install curl -y && while true; do curl https://api.telegram.org/ > /dev/null; sleep 1; done"
...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   3152      0 --:--:-- --:--:-- --:--:--  3152
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0     28      0  0:00:05  0:00:05 --:--:--    29
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0     28      0  0:00:05  0:00:05 --:--:--    29
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   2132      0 --:--:-- --:--:-- --:--:--  2735
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0     28      0  0:00:05  0:00:05 --:--:--    35
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   3452      0 --:--:-- --:--:-- --:--:--  3452

Time spent 0:00:05 indicates that there was 5 second delay for the request to complete.

Running a pod with `debian:bookworm` (no delay)

$ kubectl run curl --rm --attach -it --image=debian:bookworm --restart=Never --command -- /bin/sh -c "apt-get update && apt-get install curl -y && while true; do curl https://api.telegram.org/ > /dev/null; sleep 1; done"
...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   1672      0 --:--:-- --:--:-- --:--:--  1686
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   2128      0 --:--:-- --:--:-- --:--:--  2164
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   2123      0 --:--:-- --:--:-- --:--:--  2132
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   1973      0 --:--:-- --:--:-- --:--:--  1986
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   1943      0 --:--:-- --:--:-- --:--:--  1933
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   1892      0 --:--:-- --:--:-- --:--:--  1907
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

No delay.

Running `debian:buster` with docker on the node (no delay)

To rule out an issue with the image itself, running debian:buster directly on the node with docker. No delay.

$ sudo docker run --rm -it debian:buster /bin/sh -c "apt-get update && apt-get install curl -y && while true; do curl https://api.telegram.org/ > /dev/null; sleep 1; done"
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   2265      0 --:--:-- --:--:-- --:--:--  2265
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   3152      0 --:--:-- --:--:-- --:--:--  3152
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   2685      0 --:--:-- --:--:-- --:--:--  2685
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   2843      0 --:--:-- --:--:-- --:--:--  2843
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   3222      0 --:--:-- --:--:-- --:--:--  3152
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   145  100   145    0     0   3372      0 --:--:-- --:--:-- --:--:--  3452

femiosinowo commented 1 year ago

Disabling firewalld works for me.

mwinzie commented 1 year ago

I ran into the same issue. Just to clarify ports 0-65535 should be open on the firewall?

omidraha commented 9 months ago

I have the same issue,

The dns resolving only works on the cluster node, and not on the other nodes.

I tried fresh install with HA embedded db and also server, agent options.

The dns server address 10.43.0.10 is only accessible from cluster node ( by ssh to node) and pods on this cluster node (by running pods on this node). And is not accessible from pods on other nodes (by running pods on other nodes) or other nodes (by ssh to other nodes).

# Node 1
ubuntu@ip-10-10-1-110:~$ nmap 10.43.0.10 -Pn -p53
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:09 UTC
Nmap scan report for 10.43.0.10
Host is up (0.00015s latency).

PORT   STATE SERVICE
53/tcp open  domain

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds

# Node 2
ubuntu@ip-10-10-1-120:~$ nmap 10.43.0.10 -Pn -p53

Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:06 UTC
Nmap scan report for 10.43.0.10
Host is up.

PORT   STATE    SERVICE
53/tcp filtered domain

Nmap done: 1 IP address (1 host up) scanned in 2.04 seconds

# Node 3
ubuntu@ip-10-10-1-130:~$  nmap 10.43.0.10 -Pn -p53
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:10 UTC
Nmap scan report for 10.43.0.10
Host is up.

PORT   STATE    SERVICE
53/tcp filtered domain

Nmap done: 1 IP address (1 host up) scanned in 2.05 seconds

manuelbuil commented 9 months ago

I have the same issue,

The dns resolving only works on the cluster node, and not on the other nodes.

I tried fresh install with HA embedded db and also server, agent options.

The dns server address 10.43.0.10 is only accessible from cluster node ( by ssh to node) and pods on this cluster node (by running pods on this node). And is not accessible from pods on other nodes (by running pods on other nodes) or other nodes (by ssh to other nodes).
# Node 1
ubuntu@ip-10-10-1-110:~$ nmap 10.43.0.10 -Pn -p53
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:09 UTC
Nmap scan report for 10.43.0.10
Host is up (0.00015s latency).

PORT   STATE SERVICE
53/tcp open  domain

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds
# Node 2
ubuntu@ip-10-10-1-120:~$ nmap 10.43.0.10 -Pn -p53

Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:06 UTC
Nmap scan report for 10.43.0.10
Host is up.

PORT   STATE    SERVICE
53/tcp filtered domain

Nmap done: 1 IP address (1 host up) scanned in 2.04 seconds
# Node 3
ubuntu@ip-10-10-1-130:~$  nmap 10.43.0.10 -Pn -p53
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-29 21:10 UTC
Nmap scan report for 10.43.0.10
Host is up.

PORT   STATE    SERVICE
53/tcp filtered domain

Nmap done: 1 IP address (1 host up) scanned in 2.05 seconds

Check that your firewall allows vxlan communication. Then, verify if pod-pod communication on different nodes is working.

omidraha commented 9 months ago

Fixed, It was Firewall issue.

allnightlong commented 9 months ago

@apiening @manuelbuil thanks so much for this thread, you guys saved my day, as I run into the same issue

Gabgobie commented 9 months ago

Maybe this thread should be added to a common issues troubleshooting section in the documentation.

manuelbuil commented 9 months ago

Maybe this thread should be added to a common issues troubleshooting section in the documentation.

I'm always willing to improve our docs to avoid confusion. We are already stating that firewalld should be disabled: https://docs.k3s.io/installation/requirements?_highlight=firewall&os=rhel#operating-systems and the ports that should be open: https://docs.k3s.io/installation/requirements?_highlight=firewall&os=rhel#inbound-rules-for-k3s-server-nodes. What else would you add to help users?

Gabgobie commented 9 months ago

Good morning.

Looking at the current documentation, I think you have already done a pretty good job, especially with the first link already providing the commands to use for any given deployment.

When skimming the docs for information (and being a relative newbie such as I) the footnote about allowing all outbound ports is less noticeable. This will especially hold true for the cloud deployments some people were talking about where the firewall will be provided externally with some default settings.

I was therefore thinking of using one of the coloured fields that you already use as hints throughout the documentation to make readers aware of the fact that k3s is using the upper port ranges for its outbound container traffic and that it is not a rarity for these ports to be blocked by default, be it by an external or internal firewall.

Best, Gab

manuelbuil commented 9 months ago

Good morning.

Looking at the current documentation, I think you have already done a pretty good job, especially with the first link already providing the commands to use for any given deployment.

When skimming the docs for information (and being a relative newbie such as I) the footnote about allowing all outbound ports is less noticeable. This will especially hold true for the cloud deployments some people were talking about where the firewall will be provided externally with some default settings.

I was therefore thinking of using one of the coloured fields that you already use as hints throughout the documentation to make readers aware of the fact that k3s is using the upper port ranges for its outbound container traffic and that it is not a rarity for these ports to be blocked by default, be it by an external or internal firewall.

Best, Gab

Thanks a lot for your feedback :). The sentence k3s is using the upper port ranges for its outbound container traffic is not accurate. k3s (or kubernetes) does not use any port for external traffic starting in the cluster (e.g. in a pod). k3s (or kubernetes) just forwards whatever is sent from the pod to the outside and, normally, changes the sourceIP to match the nodeIP, but does nothing with the ports. In other words, if a pod wants to contact an internet service on $PORT, the traffic will leave the node with the $PORT field untouched

kornpow commented 5 months ago

In my case, after rebooting my machine, pods have Internet again.

k3s v1.25.4+k3s1

Ubuntu 22.04 as host OS.

Docker version 20.10.22

I feel pretty silly but rebooting solved this issue for me as well, after debugging it for a number of hours :P

I had just used the k3s install script to update from k3s 1.25.x to k3s 1.28.x and that is when the issue originally started for me.

chrisDeFouRire commented 2 months ago

I've just encountered and solved this problem thanks to this github issue:

running on Hetzner dedicated server
Linux Ubuntu-2204-jammy-amd64-base
problem appears on either k3s or microk8s (I first thought it was a microk8s problem)
no problem when running workload directly on host
pods had intermittent but frequent timeouts connecting to the internet
changing Hetzner firewall rule tcp established destination port to 0-65535 worked in real time
no more errors at all after this single modification

(I'm posting this here in the hope of helping others)

k3s-io / k3s