Open shanilhirani opened 3 months ago
fwiw, I'm seeing this same issue starting today. It was working correctly 2 days ago.
fwiw, I'm seeing this same issue starting today. It was working correctly 2 days ago.
In my case, this was solved by disconnecting from my VPN. The docker container logs pointed me toward a networking issue, which it seems to be for me.
Yeah, it's not a network issue for me as I've experienced this on two devices. Just simply rolling back seems to work without other changes so it's difficult to work out this issue.
I've had a look at changing the k3s rancher image to see if this helps but no changes in behaviour.
K3d 5.6.3 seems to something strange about how it's mapping DNS in the container.
Same here. 5.6.0 works but 5.6.2 does not (nor does 5.6.3). It seems to be using the wrong nameserver in /etc/resolv.conf inside the k3d container:
❯ k3d --version ✘ 1
k3d version v5.6.2
k3s version v1.28.8-k3s1 (default)
~/g/sandbox dev*
❯ colima ssh
me@colima:/Users/me/g/sandbox$ docker exec -it k3d-local-server-0 sh
/ # cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.
nameserver 192.168.5.2
search fritz.box
options ndots:0
# Based on host file: '/run/systemd/resolve/resolv.conf' (internal resolver)
# ExtServers: [192.168.5.2]
# Overrides: []
# Option ndots from: internal
/ # nslookup google.com
;; connection timed out; no servers could be reached
Rolling back to 5.6.0 (note how the nameserver is rewritten to 127.0.0.11):
❯ k3d --version
k3d version v5.6.0
k3s version v1.27.4-k3s1 (default)
❯ colima ssh
me@colima:/Users/me/g/sandbox$ docker exec -it k3d-local-server-0 sh
/ # cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.
nameserver 127.0.0.11
search fritz.box
options ndots:0
# Based on host file: '/run/systemd/resolve/resolv.conf' (internal resolver)
# ExtServers: [192.168.5.2]
# Overrides: []
# Option ndots from: internal
/ # nslookup google.com
Server: 127.0.0.11
Address: 127.0.0.11:53
Non-authoritative answer:
Non-authoritative answer:
Name: google.com
Address: 142.250.203.110
Fixed for me by disabling the dns fix when creating the cluster: K3D_FIX_DNS=0 k3d cluster create local
Fixed for me by disabling the dns fix when creating the cluster:
K3D_FIX_DNS=0 k3d cluster create local
@adriaanm - This workaround suggested seems to have worked.
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-6c86858495-kzdnd 1/1 Running 0 60s
kube-system coredns-6799fbcd5-mcv7r 1/1 Running 0 60s
kube-system helm-install-traefik-crd-jgxlg 0/1 Completed 0 60s
kube-system svclb-traefik-dc19675c-hm7d6 2/2 Running 0 35s
kube-system helm-install-traefik-247hw 0/1 Completed 1 60s
kube-system metrics-server-54fd9b65b-whff8 1/1 Running 0 60s
kube-system traefik-f4564c4f4-k9b8v 1/1 Running 0 35s
It would be good if was documented somewhere.
Hi,
have a look at #1445, that might describe the reason why the use of K3D_FIX_DNS=0
is a workaround for this problem.
I used this workaround K3D_FIX_DNS=0 but after stopping and restarting the cluster once it didn't seem to work anymore. Well, I didn't want to lose my experimental cluster so I dove into /etc/resolv.conf and found out that there is a wrong IP (it was the one of my colima vm). I just replaced it with my real nameserver. MacOS / colima here btw
What did you do
How was the cluster created?
k3d cluster create mycluster
What did you do afterwards?
What did you expect to happen
I was expecting a deploy of nginx to be started and ready for consumption however upon investigation is appears that
mycluster
does not appear to start correctly as pods stuck in acontainercreating
state, they seem to be failing to pull down container images.NOTE: This issue DOES NOT OCCUR when using K3d 5.6.0, as I have rolled back to this version and the cluster bootstraps fine.
Screenshots or terminal output
Which OS & Architecture
k3d runtime-info
Which version of
k3d
k3d version
Which version of docker
docker version
anddocker info
Server: Docker Engine - Community Engine: Version: 26.1.1 API version: 1.45 (minimum version 1.24) Go version: go1.21.9 Git commit: ac2de55 Built: Tue Apr 30 11:48:47 2024 OS/Arch: linux/arm64 Experimental: false containerd: Version: 1.6.31 GitCommit: e377cd56a71523140ca6ae87e30244719194a521 runc: Version: 1.1.12 GitCommit: v1.1.12-0-g51d5e94 docker-init: Version: 0.19.0 GitCommit: de40ad0
Client: Docker Engine - Community Version: 26.1.3 Context: colima Debug Mode: false Plugins: compose: Docker Compose (Docker Inc.) Version: 2.27.1 Path: /Users/$USER/.docker/cli-plugins/docker-compose
Server: Containers: 3 Running: 3 Paused: 0 Stopped: 0 Images: 8 Server Version: 26.1.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: e377cd56a71523140ca6ae87e30244719194a521 runc version: v1.1.12-0-g51d5e94 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.8.0-31-generic Operating System: Ubuntu 24.04 LTS OSType: linux Architecture: aarch64 CPUs: 2 Total Memory: 1.91GiB Name: colima ID: a09eda6a-75aa-4810-960d-0718469dc07d Docker Root Dir: /var/lib/docker Debug Mode: false Username: $USER Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false