Can't access external website with local Kubernetes in docker-desktop 4.25.0 releae

cdlliuy commented 10 months ago

Description

with the latest https://docs.docker.com/desktop/release-notes/#4250 4.25.0 release, our application running on the local Kubernetes can't access external website anymore.

The coredns pods kept failing with

[ERROR] plugin/errors: 2 xxx. AAAA: dns: overflowing header size
[ERROR] plugin/errors: 2 xxx A: dns: overflow unpacking uint16

We already identified it is a coredns issue with https://github.com/coredns/coredns/issues/5998, and can be migrated by downgrade the coredns version to v1.10.0

can the upstream docker-desktop fix the coredns version issue asap with a new release?

Reproduce

deploy any container to local kuberenets hosted by docker

do curl cmd inside the container:


curl https://login.microsoftonline.com -v

Could not resolve host: login.microsoftonline.com
Closing connection 0 curl: (6) Could not resolve host: login.microsoftonline.com
Check coredns logs: [ERROR] plugin/errors: 2 login.microsoftonline.com. A: dns: overflow unpacking uint16 [ERROR] plugin/errors: 2 login.microsoftonline.com. AAAA: dns: overflowing header size

Expected behavior

No response

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:48 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.25.0 (126437)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:16 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

N/A

Diagnostics ID

N/A

Additional Info

No response

JoFrMueller commented 10 months ago

For us it's similar though doesn't even have to be K8s.

/ # curl https://api.nuget.org/ -v
* processing: https://api.nuget.org/
* Could not resolve host: api.nuget.org
* Closing connection
curl: (6) Could not resolve host: api.nuget.org

Interestingly enough many others like google work:

/ # curl https://google.com/ -v
* processing: https://google.com/
*   Trying 142.251.36.174:443...
* Connected to google.com (142.251.36.174) port 443
* ALPN: offers h2,http/1.1

Going back to 4.24.4 all is good again...

aarondonohue commented 9 months ago

I too have encountered this issue. Downgrading from 4.26.1 to 4.24.2 solved my problem.

rsheasby commented 8 months ago

Same issue. Using a 2 month old version of Docker Desktop is sub-optimal, has anyone found a workaround?

PhilHamlin-MS commented 8 months ago

That issue is caused by this issuehttps://github.com/coredns/coredns/issues/5998, so our workaround was a DNS solution not involving CoreDNS for now.

From: Ryan David Sheasby @.> Sent: Monday, January 8, 2024 1:20 AM To: docker/for-win @.> Cc: Manual @.***> Subject: Re: [docker/for-win] Can't access external website with local Kubernetes in docker-desktop 4.25.0 releae (Issue #13768)

Same issue. Using a 2 month old version of Docker Desktop is sub-optimal, has anyone found a workaround?

— Reply to this email directly, view it on GitHubhttps://github.com/docker/for-win/issues/13768#issuecomment-1880630777 or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAHGXTGCIVFMQLX2PY6B6NLYNO24NBFKMF2HI4TJMJ2XIZLTS2BKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVE2DMMZYGMZTENRRURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2SOBSGUYDANBTGU2KI3TBNVS2S2DBONPWYYLCMVWIFJLWMFWHKZNKGYYTGMZWGM2TQMRVURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHFMCUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2QNRUGQYDKOBXGWBKI5DZOBS2K2LTON2WLJLWMFWHKZNKGE4TMNZZGEZTKMJUQKSHI6LQMWSWYYLCMVWKK5TBNR2WLKJUGYZTQMZTGI3DDAVEOR4XAZNFNRQWEZLMUV3GC3DVMWUTQMRVGAYDIMZVGSBKI5DZOBS2K3DBMJSWZJLWMFWHKZNKGYYTGMZWGM2TQMRVU52HE2LHM5SXFJTDOJSWC5DF. You are receiving this email because you are subscribed to this thread.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

jasase commented 7 months ago

Does some one know if this problem is solved in Version 4.27.0?

PhilHamlin-MS commented 7 months ago

The GitHub issue's still open. (Apologies, I can't tell whether that's you asking exactly this question on GitHub.) Can't access external website with local Kubernetes in docker-desktop 4.25.0 releae · Issue #13768 · docker/for-win (github.com)https://github.com/docker/for-win/issues/13768

From: Sascha Sternheim @.> Sent: Sunday, January 28, 2024 10:17 PM To: docker/for-win @.> Cc: Comment @.>; Manual @.> Subject: Re: [docker/for-win] Can't access external website with local Kubernetes in docker-desktop 4.25.0 releae (Issue #13768)

Does some one know if this problem is solved in Version 4.27.0?

— Reply to this email directly, view it on GitHubhttps://github.com/docker/for-win/issues/13768#issuecomment-1914031641 or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAHGXTFSP3TSLNNYKIXSXSLYQ45G5BFKMF2HI4TJMJ2XIZLTS2BKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVE2DMMZYGMZTENRRURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2SOBSGUYDANBTGU2KI3TBNVS2S2DBONPWYYLCMVWIFJLWMFWHKZNKGYYTGMZWGM2TQMRVURXGC3LFVFUGC427NRQWEZLMVRZXKYTKMVRXIX3UPFYGLLCJONZXKZKDN5WW2ZLOOSTHI33QNFRXHFMCUR2HS4DFVJZGK4DPONUXI33SPGSXMYLMOVS2QNRUGQYDKOBXGWBKI5DZOBS2K2LTON2WLJLWMFWHKZNKGE4TMNZZGEZTKMJUQKSHI6LQMWSWYYLCMVWKK5TBNR2WLKJUGYZTQMZTGI3DDAVEOR4XAZNFNRQWEZLMUV3GC3DVMWUTQMRVGAYDIMZVGSBKI5DZOBS2K3DBMJSWZJLWMFWHKZNKGYYTGMZWGM2TQMRVU52HE2LHM5SXFJTDOJSWC5DF. You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

pcatlin commented 7 months ago

No, still broken for me using 4.27.0 - seems like all OS's are affected too - Mac #7110

dennis-yemelyanov commented 7 months ago

any plans to fix?

crowne commented 7 months ago

As @cdlliuy says, this can be resolved by rolling coredns back from v1.10.1 to v1.10.0

here's the patch command:

kubectl patch deployment coredns -n kube-system -p '{"spec":{"template":{"spec":{"containers":[{"name":"coredns","image":"registry.k8s.io/coredns/coredns:v1.10.0"}]}}}}'

bogdanghervan commented 6 months ago

Still broken in 4.28.0.

qcaas-nhs-sjt commented 6 months ago

I've tried rolling back to version 1.10.0 on docker desktop, however this issue is still occurring for me. When can we expect a proper fix for this?

JoFrMueller commented 6 months ago

@qcaas-nhs-sjt - we run on Docker Desktop 4.24.2 like @aarondonohue mentioned since ages felt, works very well for us. Just never upgrade - juniors will rage, but we had 10 erroneous upgrade attempts, so we just don't care anymore for and choose that version only. It's only dev machines anyways in our case, - so we're lucky there.

Ensure to have uninstalled Docker Desktop properly before installing the old version, then all should be fine - at least for docker compose, K8s I don't have any clue - sorry.

qcaas-nhs-sjt commented 6 months ago

@qcaas-nhs-sjt - we run on Docker Desktop 4.24.2 like @aarondonohue mentioned since ages felt, works very well for us. Just never upgrade - juniors will rage, but we had 10 erroneous upgrade attempts, so we just don't care anymore for and choose that version only. It's only dev machines anyways in our case, - so we're lucky there.

Ensure to have uninstalled Docker Desktop properly before installing the old version, then all should be fine - at least for docker compose, K8s I don't have any clue - sorry.

Thanks for this, I understand that this is a suggested workaround but we are now on v4.28.0, I have a team of people that are new to kubernetes that I am building a framework for and many of these new developers will be using windows machines in a windows environment where the centrally managed IT team will be wanting to ensure the environment is properly patched for security issues. As a result we must provide an environment that is updateable. There are currently security fixes in the patches since so it is really not suitable in our environment so a fix for this is needed and while we can work around it in this way we really shouldn't be.

jasase commented 5 months ago

any plans to fix? Seems there is no progress on this topic.

KavyaShree25 commented 4 months ago

i m unable to install previous versions of docker desktop.any help here

patrickhuber commented 3 months ago

I managed to use a coredns config map to work around this issue. Sending all requests to the login.microsoftonline.com domain to cloudflare 1.1.1.1

coredns.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    login.microsoftonline.com {
      forward . 1.1.1.1
    }

    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

kubectl apply -f coredns.yml

Here is the documentation on patching dns

https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#are-dns-queries-being-received-processed

Here is the documentation on coredns forward entries

https://coredns.io/plugins/forward/

I exported my config using the following command before adding in the forward section on login.microsoftonline.com

kubectl -n kube-system get configmap coredns

It appears coredns 1.11.3 fixes a UDP DNS Overflow but it isn't available to downstream kubernetes as a container yet so this is temporary until the new image is available. https://github.com/coredns/coredns/releases/tag/v1.11.3

You will get a warning about missing annotations. I think this is because I didn't edit the config, I exported it and re-imported it. kubectl fixes it so I guess its not too much of an issue.

docker / for-win