search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
26.71k stars 2.24k forks source link

Image pulls from embedded registry fail if --bind-address is set (and not 127.0.0.1) #10094

Open achristianson opened 2 weeks ago

achristianson commented 2 weeks ago

Environmental Info: K3s Version: v1.30.0+k3s1

Node(s) CPU architecture, OS, and Version: Linux viking 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux

Cluster Configuration: 3 servers 4 agents with embedded registry enabled

Describe the bug:

When I have --bind-address set on my masters, pods scheduled on the masters cannot pull from 127.0.0.1:6443 because it isn't bound to 127.0.0.1 --it's bound to a different IP.

Steps To Reproduce:

Set up a cluster with embedded registry. Set --bind-address to anything other than 127.0.0.1.

Expected behavior:

When a pod is scheduled, k3s is expected to pull from the correct address when --bind-address is set.

Actual behavior:

When a pod is scheduled, k3s tries to pull from 127.0.0.1 but since k3s is bound to a different IP, it gets connection refused.

Additional context / logs:

brandond commented 2 weeks ago

I can take a look at addressing this when I bump the embedded spegel version. I will note that this currently only affects servers, as agents do not support the --bind-address flag, so the listener address cannot be configured. Might want to promote that to an agent flag as well I guess.

achristianson commented 2 weeks ago

Might want to promote that to an agent flag as well I guess.

Having the bind flag on agents would definitely be desirable (and make the overall config options more consistent).

brandond commented 2 weeks ago

there are already some changes staged in this space in

brandond commented 2 weeks ago

One of the problems here is that spegel needs to be able to identify which requests are from the local containerd instance and should proxy to other nodes, as opposed to those from other nodes that should not be reproxied. Right now this is done by identifying requests to localhost, and I can't think of another really good way to do that without opening more ports. We may just need to set it up so that it binds to localhost plus the configured bind address.

brandond commented 2 weeks ago

OK, I've modified that PR to always bind the supervisor on the loopback addresses, in addition to the requested address

brandond commented 3 days ago

Note that the embedded registry still will not work on nodes where IPv6 is the primary address family due to https://github.com/k3s-io/k3s/issues/9897