Can't resolve oauth2.googleapis.com in kubernetes-Pod / CoreDNS-Bug

adorn commented 8 months ago

Description

Can't resolve DNS-Name oauth2.googleapis.com inside a Kubernetes-Pod Other hostnames are working.

Reproduce

Start Kubernetes inside Docker-App
Run a pod
Loginto a pod kubectl exec -it pod/mypod -- sh

wget oauth2.googleapis.com. returns:

Resolving oauth2.googleapis.com (oauth2.googleapis.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘oauth2.googleapis.com’

Run kubectl logs coredns-5dd5756b68-5cstw --namespace=kube-system. returns:

CoreDNS-1.10.1
linux/arm64, go1.20, 055b2c3
[ERROR] plugin/errors: 2 oauth2.googleapis.com. A: dns: overflow unpacking uint32
...

Expected behavior

Getting a IP-Adress for oauth2.googleapis.com (and Reponse.)

docker version

docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.25.2 (129061)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/amenze/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.0-desktop.1
    Path:     /Users/amenze/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/amenze/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/amenze/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.9
    Path:     /Users/amenze/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/amenze/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/amenze/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.9
    Path:     /Users/amenze/.docker/cli-plugins/docker-scout

Server:
 Containers: 31
  Running: 30
  Paused: 0
  Stopped: 1
 Images: 20
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 23.44GiB
 Name: linuxkit-566a8df0b5b5
 ID: bdb7decc-ffb1-430a-9bca-229a8c838c08
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

Diagnosing...

Additional Info

Downgrading to CoreDNS 1.10.0 heps. See also: https://github.com/coredns/coredns/issues/6431

My Docker-App was not able to give me a Diagnostics ID. for 20 minutes Still Diagnosing...

jordiclariana commented 8 months ago

We are experiencing the same, with 4.25.2 and now also with 4.26.0. This started happening a couple of days ago and it is intermittent. Sometimes, after a long time, it manages to resolve the host oauth2.googleapis.com.

xvzf commented 8 months ago

Small update on this one -- it seems the docker-desktop internal DNS resolver/proxy/? is not supporting DNS compression (see more on rfc1035) thus bloating the DNS response over the limit for the UDP datagram (512 bytes), which is enforced by CoreDNS.

Here are two packets, one coming from my router, the other one captured targeting the CoreDNS pod in docker-desktop. The difference is pretty clear

0000   d8 ec 5e bc ad ed ae f5 7c ec 1d 70 08 00 45 00
0010   01 43 7c 39 40 00 40 11 3b 55 c0 a8 00 01 c0 a8
0020   00 ca 00 35 f1 3f 01 2f 69 f2 9f c6 81 80 00 01
0030   00 10 00 00 00 00 06 6f 61 75 74 68 32 0a 67 6f
0040   6f 67 6c 65 61 70 69 73 03 63 6f 6d 00 00 01 00
0050   01 c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa ba
0060   2a c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa b9
0070   aa c0 0c 00 01 00 01 00 00 00 83 00 04 ac d9 12
0080   0a c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa ba
0090   8a c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa ba
00a0   6a c0 0c 00 01 00 01 00 00 00 83 00 04 d8 3a ce
00b0   2a c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa b9
00c0   8a c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa ba
00d0   4a c0 0c 00 01 00 01 00 00 00 83 00 04 ac d9 10
00e0   ca c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa b9
00f0   ea c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa b9
0100   6a c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa b5
0110   ea c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa b9
0120   ca c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa b9
0130   4a c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa ba
0140   aa c0 0c 00 01 00 01 00 00 00 83 00 04 8e fa 4a
0150   ca

And the packet generated by docker-desktop internal DNS resolver/proxy/? (still not sure what it is)

0000   7a 73 1c a9 0f 07 46 e2 65 5d fb 78 08 00 45 00
0010   02 93 e6 5e 40 00 3f 11 46 cf c0 a8 41 07 0a 01
0020   00 7c 00 35 a0 d1 02 7f 0e bd 97 72 81 80 00 01
0030   00 10 00 00 00 00 06 6f 61 75 74 68 32 0a 67 6f
0040   6f 67 6c 65 61 70 69 73 03 63 6f 6d 00 00 01 00
0050   01 06 6f 61 75 74 68 32 0a 67 6f 6f 67 6c 65 61
0060   70 69 73 03 63 6f 6d 00 00 01 00 01 00 00 01 30
0070   00 04 ac d9 12 0a 06 6f 61 75 74 68 32 0a 67 6f
0080   6f 67 6c 65 61 70 69 73 03 63 6f 6d 00 00 01 00
0090   01 00 00 01 30 00 04 ac d9 10 ca 06 6f 61 75 74
00a0   68 32 0a 67 6f 6f 67 6c 65 61 70 69 73 03 63 6f
00b0   6d 00 00 01 00 01 00 00 01 30 00 04 8e fa b8 ea
00c0   06 6f 61 75 74 68 32 0a 67 6f 6f 67 6c 65 61 70
00d0   69 73 03 63 6f 6d 00 00 01 00 01 00 00 01 30 00
00e0   04 ac d9 10 8a 06 6f 61 75 74 68 32 0a 67 6f 6f
00f0   67 6c 65 61 70 69 73 03 63 6f 6d 00 00 01 00 01
0100   00 00 01 30 00 04 8e fa ba 8a 06 6f 61 75 74 68
0110   32 0a 67 6f 6f 67 6c 65 61 70 69 73 03 63 6f 6d
0120   00 00 01 00 01 00 00 01 30 00 04 8e fa ba 6a 06
0130   6f 61 75 74 68 32 0a 67 6f 6f 67 6c 65 61 70 69
0140   73 03 63 6f 6d 00 00 01 00 01 00 00 01 30 00 04
0150   8e fa ba 2a 06 6f 61 75 74 68 32 0a 67 6f 6f 67
0160   6c 65 61 70 69 73 03 63 6f 6d 00 00 01 00 01 00
0170   00 01 30 00 04 ac d9 17 6a 06 6f 61 75 74 68 32
0180   0a 67 6f 6f 67 6c 65 61 70 69 73 03 63 6f 6d 00
0190   00 01 00 01 00 00 01 30 00 04 8e fa b8 ca 06 6f
01a0   61 75 74 68 32 0a 67 6f 6f 67 6c 65 61 70 69 73
01b0   03 63 6f 6d 00 00 01 00 01 00 00 01 30 00 04 d8
01c0   3a d4 8a 06 6f 61 75 74 68 32 0a 67 6f 6f 67 6c
01d0   65 61 70 69 73 03 63 6f 6d 00 00 01 00 01 00 00
01e0   01 30 00 04 8e fa b9 4a 06 6f 61 75 74 68 32 0a
01f0   67 6f 6f 67 6c 65 61 70 69 73 03 63 6f 6d 00 00
0200   01 00 01 00 00 01 30 00 04 8e fa b5 ea 06 6f 61
0210   75 74 68 32 0a 67 6f 6f 67 6c 65 61 70 69 73 03
0220   63 6f 6d 00 00 01 00 01 00 00 01 30 00 04 d8 3a
0230   ce 2a 06 6f 61 75 74 68 32 0a 67 6f 6f 67 6c 65
0240   61 70 69 73 03 63 6f 6d 00 00 01 00 01 00 00 01
0250   30 00 04 8e fa ba 4a 06 6f 61 75 74 68 32 0a 67
0260   6f 6f 67 6c 65 61 70 69 73 03 63 6f 6d 00 00 01
0270   00 01 00 00 01 30 00 04 ac d9 12 6a 06 6f 61 75
0280   74 68 32 0a 67 6f 6f 67 6c 65 61 70 69 73 03 63
0290   6f 6d 00 00 01 00 01 00 00 01 30 00 04 8e fa ba
02a0   aa

remusmp commented 8 months ago

I came across the same issue:

[INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2 │
│ CoreDNS-1.10.1                                                                                                     │
│ linux/amd64, go1.20, 055b2c3                                                                                       │
│ [ERROR] plugin/errors: 2 oauth2.googleapis.com. AAAA: read udp 10.1.34.108:39322->192.168.65.7:53: i/o timeout     │
│ [ERROR] plugin/errors: 2 oauth2.googleapis.com. AAAA: dns: overflow unpacking uint16

docker-desktop 4.26.1 docker-desktop kubernetes 1.28.2

I also tried uprgading coredns to 1.11.1

│ [INFO] plugin/reload: Running configuration SHA512 = 591cf328cccc12bc490481273e738df59329c62c0b729d94e8b61db9961c2 │
│ CoreDNS-1.11.1                                                                                                     │
│ linux/amd64, go1.20.7, ae2bbc2                                                                                     │
│ [ERROR] plugin/errors: 2 login.microsoftonline.com. A: read udp 10.1.34.124:54184->192.168.65.7:53: i/o timeout    │
│ [ERROR] plugin/errors: 2 login.microsoftonline.com. A: dns: overflow unpacking uint16

It works fine on my setup with CoreDNS-1.9.4.