[QUESTION/HELP] DNSError when switching to mobile network

schlichtanders commented 11 months ago

I would like to use k3d to work on the go, but when switching to mobile network, the cluster cannot resolve DNS. A similar http request from normal terminal does not run into this.

I've read about https://github.com/k3d-io/k3d/issues/209 but it didn't help in my case

What did you do

How was the cluster created?
- k3d cluster create --registry-use k3d-registry.localhost:12345 --api-port 6550 -p "80:80@loadbalancer" --image "rancher/k3s:v1.26.9-k3s1" --servers 1 --agents 1
What did you do afterwards?
- k3d commands? no further k3d commands
- docker commands? I was deploying my application to the cluster
- OS operations (e.g. shutdown/reboot)? I switched to mobile network

What did you expect to happen

K3d should be able to resolve DNS without error also after switching to mobile network. The browser and terminal do well, but applications running inside the k3d cluster fail to resolve DNS.

Screenshots or terminal output

DNSError: api.github.com, temporary failure (EAI_AGAIN)

My program tries to access the api.github.com server

Which OS & Architecture

output of k3d runtime-info

arch: x86_64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: gram17
name: docker
os: NixOS 23.05 (Stoat)
ostype: linux
version: 20.10.25

Which version of `k3d`

output of k3d version

k3d version v5.6.0
k3s version v1.21.7-k3s1 (default)

Which version of docker

output of docker version and docker info


Client:
Version:           20.10.25
API version:       1.41
Go version:        go1.20.8
Git commit:        v20.10.25
Built:             Thu Jan  1 00:00:00 1970
OS/Arch:           linux/amd64
Context:           default
Experimental:      true

Server: Engine: Version: 20.10.25 API version: 1.41 (minimum version 1.12) Go version: go1.20.8 Git commit: v20.10.25 Built: Tue Jan 1 00:00:00 1980 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.7.1 GitCommit: v1.7.1 runc: Version: 1.1.5 GitCommit:
docker-init: Version: 0.19.0 GitCommit:

Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc., v0.10.4) compose: Docker Compose (Docker Inc., 2.18.1)

Server: Containers: 5 Running: 4 Paused: 0 Stopped: 1 Images: 83 Server Version: 20.10.25 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: journald Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: v1.7.1 runc version: init version: Security Options: seccomp Profile: default cgroupns Kernel Version: 6.5.3 Operating System: NixOS 23.05 (Stoat) OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.26GiB Name: gram17 ID: (NOT SURE WHETHER THIS IS SENSIBLE INFORMATION - PLEASE DON'T LET PEOPLE SHARE SENSITIVE INFORMATION) Docker Root Dir: /var/lib/docker Debug Mode: false Username: THIS IS PROBABLY SENSIBLE INFORMATION - PLEASE DON'T LET PEOPLE SHARE SENSITIVE INFORMATION Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: true

schlichtanders commented 11 months ago

When enabling export K3D_FIX_DNS=1, the cert-manager (and probably others too) do not find rancher/mirrored-pause.

kubectl describe pod cert-manager-webhook -n cert-manager prints

...
 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": failed to do request: Head "https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6": dial tcp 18.215.138.58:443: i/o timeout
...

schlichtanders commented 11 months ago

A guess: my mobile network does not seem to find a couple of ip4 addresses. My host laptop resolves registry-1.docker.io not to the above mentioned ip4, but only to ip6 addresses.

EDIT: If I try to reach the ip addresses printed inside k3d from a laptop top-level terminal, I indeed cannot reach this IP.

Could this be the issues? Does k3d not support ip6 routing?

schlichtanders commented 11 months ago

Further research brought me to this post which explains that my mobile provider (deutsche telekom) uses a new DNS resolution system which translates ip4 only addresses to some fancy ip6 wrapper which are then again unwrapped by the provider.

this DNS64 resolution does not seem to work when setting K3D_FIX_DNS=1 ... need to investigate further...

k3d-io / k3d