The coredns container is endlessly restarting

[metacoma]

Container logs:

root@minikube:/# docker logs -f c015cf083060
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration SHA512 = 05e3eaddc414b2d71a69b2e2bc6f2681fc1f4d04bcdd3acc1a41457bb7db518208b95ddfc4c9fffedc59c25a8faf458be1af4915a4a3c0d6777cb7a346bc5d86
CoreDNS-1.10.1
linux/amd64, go1.20, 055b2c3
[INFO] 127.0.0.1:50302 - 14823 "HINFO IN 7953148733213368499.4453155829133784578. udp 57 false 512" - - 0 6.003067196s
[ERROR] plugin/errors: 2 7953148733213368499.4453155829133784578. HINFO: read udp 10.244.0.2:36315->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:42987 - 13272 "HINFO IN 7953148733213368499.4453155829133784578. udp 57 false 512" - - 0 6.002698336s
[ERROR] plugin/errors: 2 7953148733213368499.4453155829133784578. HINFO: read udp 10.244.0.2:50808->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:47092 - 59248 "HINFO IN 7953148733213368499.4453155829133784578. udp 57 false 512" - - 0 4.001190974s
[ERROR] plugin/errors: 2 7953148733213368499.4453155829133784578. HINFO: read udp 10.244.0.2:37685->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:45686 - 48752 "HINFO IN 7953148733213368499.4453155829133784578. udp 57 false 512" - - 0 2.001175282s
[ERROR] plugin/errors: 2 7953148733213368499.4453155829133784578. HINFO: read udp 10.244.0.2:40673->192.168.49.1:53: i/o timeout
[INFO] 127.0.0.1:49354 - 65204 "HINFO IN 7953148733213368499.4453155829133784578. udp 57 false 512" - - 0 2.000775685s
[ERROR] plugin/errors: 2 7953148733213368499.4453155829133784578. HINFO: read udp 10.244.0.2:44471->192.168.49.1:53: i/o timeout
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: i/o timeout
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: i/o timeout
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 5s

Steps to reproduce:

terraform {
        required_providers {
                minikube = {
                        source = "scott-the-programmer/minikube"
                        version = "0.3.7"
                } 
        }

}

provider minikube {
  kubernetes_version = "v1.28.3"
}

resource "minikube_cluster" "cluster" {
  driver       = "docker"
  cluster_name = "minikube"
  addons = [
    "default-storageclass",
    "storage-provisioner",
    "dashboard",
    "ingress"
  ]
  wait = ["all"]
  host_dns_resolver = false
  memory = "16384mb"
}

What's interesting is that when deploying the cluster using minikube without terraform, everything works as expected.

$ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d
$ /usr/local/bin/minikube start --profile minikube2 --driver=docker 
$ docker exec -ti minikube2 docker logs 8229446a5a47
.:53
[INFO] plugin/reload: Running configuration SHA512 = 75e5db48a73272e2c90919c8256e5cca0293ae0ed689e2ed44f1254a9589c3d004cb3e693d059116718c47e9305987b828b11b2735a1cefa59e4a9489dda5cee
CoreDNS-1.10.1
linux/amd64, go1.20, 055b2c3
[INFO] 127.0.0.1:58269 - 9097 "HINFO IN 8133781860100051641.7404535210535734537. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.01729004s

[metacoma]

It seems I have found the solution to the connectivity issue for minikube deployed using terraform resources.

After several hours of searching on Google, I came across this Slack thread with similar symptoms: https://slack-archive.rancher.com/t/10289479/hi-since-a-few-hours-ago-my-dns-in-k3s-stopped-working-nobod#538f20db-d53f-4443-8bc7-6f414988ebfe

So, after executing the following commands:

ufw allow 6443/tcp # 1. This would expose my API server to the internet. Not going to do this.
ufw allow from 10.42.0.0/16 to any # 2. Allow all pods to communicate with my host.
ufw allow from 10.43.0.0/16 to any # 3. Allow all services to communicate with my host.

coredns works fine

The unanswered question is: why does the kubernetes cluster deployed directly using minikube have no issue with the CoreDNS pod, while minikube deployed using terraform encounters this problem?

Both k8s clusters I deployed are on the same machine.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

Anyway, feel free to close this issue.

[metacoma]

The unanswered question is: why does the kubernetes cluster deployed directly using minikube have no issue with the CoreDNS pod, while minikube deployed using terraform encounters this problem?

It appears that the root cause of this issue is that I ran terraform inside a docker container.