search

kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.04k stars 1.51k forks source link

coredns pod can't connected the kind bridge network HINFO: read udp 10.244.0.3:49899->172.18.0.1:53: i/o timeout #3550

Open LauYoung7 opened 3 months ago

LauYoung7 commented 3 months ago

What happened: my kind cluster corrdns pod reported this error:

[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2fa5f046dd37f1cf888b953814040d180f52594972691cd6ff41be96639138a43908 CoreDNS-1.10.1 linux/amd64, go1.20, 055b2c3 plugin/errors: 2 620473533967225528.5223075680251020783. HINFO: read udp 10.244.0.3:49899->172.18.0.1:53: i/o timeout [ERROR] plugin/errors: 2 620473533967225528.5223075680251020783. HINFO: read udp 10.244.0.3:48078->172.18.0.1:53: i/o timeout [ERROR] plugin/errors: 2 620473533967225528.5223075680251020783. HINFO: read udp 10.244.0.3:41001->172.18.0.1:53: i/o timeout

What you expected to happen: the coredns pod should work correctly.

How to reproduce it (as minimally and precisely as possible): Here is my kind cluster config cat << EOF | kind create cluster --config=- kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking:

the default CNI will not be installed

disableDefaultCNI: true nodes:

Anything else we need to know?: 屏幕截图 2024-03-13 182432 I used the flannel plugin instead of the default cni of kind image

aojea commented 3 months ago

it seems your cni or something you are adding to the nodes is not able to forward correctly the dns queries

stmcginnis commented 3 months ago 
disableDefaultCNI: true

~Make sure you are installing a CNI.~ Update: Sorry, I missed your screenshot showing Flannel.

Also see #3353

stmcginnis commented 3 months ago

Please add the output from docker info. That contains a lot of system details that can help troubleshooting.

If possible, please don't paste screenshots of text output. Copy the output and place inside a codeblock. In markdown this is done by starting with three backticks (`), then the content, then closing the codeblock with another three backticks.

/remove-kind bug /kind support

LauYoung7 commented 3 months ago

Here is the docker info

[root@bclinux ~]# docker info Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Docker Buildx (Docker Inc., v0.7.1-docker) scan: Docker Scan (Docker Inc., v0.23.0)

Server: Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 24 Server Version: 20.10.12 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523 runc version: v1.1.9-0-gccaecfc init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 5.10.0-136.49.0.127.10.oe2203.bclinux.x86_64 Operating System: BigCloud Enterprise Linux For Euler 22.10 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 30.77GiB Name: bclinux ID: TDUJ:P2R4:7UCM:5QXV:S4T4:SMR3:Z7DU:N2DT:A7S4:ZA4Y:4BYN:ER7D Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http://172.31.192.47:6687 HTTPS Proxy: http://172.31.192.47:6687 No Proxy: localhost,127.0.0.1,.example.com Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Registry Mirrors: https://eu437ruz.mirror.aliyuncs.com/ Live Restore Enabled: false

LauYoung7 commented 3 months ago

Here is something my network config and the container network config VM: 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:3e:b7:60:28 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: br-8717e7c72efa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:46:c0:a9:60 brd ff:ff:ff:ff:ff:ff inet 172.18.0.1/16 brd 172.18.255.255 scope global br-8717e7c72efa valid_lft forever preferred_lft forever inet6 fc00:f853:ccd:e793::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::42:46ff:fec0:a960/64 scope link valid_lft forever preferred_lft forever inet6 fe80::1/64 scope link valid_lft forever preferred_lft forever 6: vethbdd7255@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-8717e7c72efa state UP group default link/ether ce:c7:f5:52:2f:c7 brd ff:ff:ff:ff:ff:ff link-netnsid 0

container: [root@bclinux ~]# docker exec -it kind-control-plane cat /etc/resolv.conf nameserver 172.18.0.1 options ndots:0 [root@bclinux ~]# docker exec -it kind-control-plane ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default link/ether e2:14:ac:5e:00:c9 brd ff:ff:ff:ff:ff:ff inet 10.244.0.0/32 scope global flannel.1 valid_lft forever preferred_lft forever 3: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000 link/ether f6:e2:1d:1c:32:9b brd ff:ff:ff:ff:ff:ff inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0 valid_lft forever preferred_lft forever 4: veth704b1078@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default link/ether f6:e2:1d:1c:32:9b brd ff:ff:ff:ff:ff:ff link-netns cni-8797824d-770d-dfad-deea-cd92d8a77f52 5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fc00:f853:ccd:e793::2/64 scope global nodad valid_lft forever preferred_lft forever inet6 fe80::42:acff:fe12:2/64 scope link valid_lft forever preferred_lft forever 6: vethf3b8c944@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default link/ether b6:d2:2e:95:cf:69 brd ff:ff:ff:ff:ff:ff link-netns cni-01fb463b-b628-d1f9-c61c-686ff0cd381b 7: veth28211424@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default link/ether 7a:79:fd:df:31:9c brd ff:ff:ff:ff:ff:ff link-netns cni-18fcc57d-c067-e376-8d6f-a381c0ec9021