Open cwilkers opened 3 years ago
@cwilkers we've done some work to fix networking for multinode clusters, can you try again this with the newest version of minikube and see if it still persists?
Unfortunately, I see the same behavior with v1.22.0.
$ minikube version
minikube version: v1.22.0
commit: a03fbcf166e6f74ef224d4a63be4277d017bb62e
$ minikube start --cni=flannel --nodes=2
😄 minikube v1.22.0 on Fedora 34
✨ Automatically selected the kvm2 driver. Other choices: podman, none, ssh
💾 Downloading driver docker-machine-driver-kvm2:
> docker-machine-driver-kvm2....: 65 B / 65 B [----------] 100.00% ? p/s 0s
> docker-machine-driver-kvm2: 11.47 MiB / 11.47 MiB 100.00% 343.09 MiB p/s
💿 Downloading VM boot image ...
> minikube-v1.22.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s
> minikube-v1.22.0.iso: 242.95 MiB / 242.95 MiB 100.00% 98.80 MiB p/s 2.7s
👍 Starting control plane node minikube in cluster minikube
💾 Downloading Kubernetes v1.21.2 preload ...
> preloaded-images-k8s-v11-v1...: 502.14 MiB / 502.14 MiB 100.00% 84.82 Mi
🔥 Creating kvm2 VM (CPUs=2, Memory=2200MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.21.2 on Docker 20.10.6 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring Flannel (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
👍 Starting node minikube-m02 in cluster minikube
🔥 Creating kvm2 VM (CPUs=2, Memory=2200MB, Disk=20000MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.39.50
🐳 Preparing Kubernetes v1.21.2 on Docker 20.10.6 ...
▪ env NO_PROXY=192.168.39.50
🔎 Verifying Kubernetes components...
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ kubectl get po -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-558bd4d5db-hvnqx 1/1 Running 0 60s 10.88.0.2 minikube <none> <none>
kube-system etcd-minikube 1/1 Running 0 67s 192.168.39.50 minikube <none> <none>
kube-system kube-apiserver-minikube 1/1 Running 0 67s 192.168.39.50 minikube <none> <none>
kube-system kube-controller-manager-minikube 1/1 Running 0 67s 192.168.39.50 minikube <none> <none>
kube-system kube-flannel-ds-amd64-4sbx9 1/1 Running 0 61s 192.168.39.50 minikube <none> <none>
kube-system kube-flannel-ds-amd64-h7h58 1/1 Running 0 33s 192.168.39.236 minikube-m02 <none> <none>
kube-system kube-proxy-698zf 1/1 Running 0 33s 192.168.39.236 minikube-m02 <none> <none>
kube-system kube-proxy-z2r66 1/1 Running 0 61s 192.168.39.50 minikube <none> <none>
kube-system kube-scheduler-minikube 1/1 Running 0 67s 192.168.39.50 minikube <none> <none>
kube-system storage-provisioner 1/1 Running 0 73s 192.168.39.50 minikube <none> <none>
$ minikube addons enable kubevirt
▪ Using image bitnami/kubectl:1.17
🌟 The 'kubevirt' addon is enabled
$ kubectl -n kube-system get po kubevirt-install-manager -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kubevirt-install-manager 1/1 Running 0 5m10s 10.244.1.2 minikube-m02 <none> <none>
$ kubectl -n kube-system logs kubevirt-install-manager
error: the path "/manifests/kubevirt-base.yaml" does not exist
error: the path "/manifests/kubevirt.yaml" does not exist
$ kubectl -n kube-system exec -ti kubevirt-install-manager bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
I have no name!@kubevirt-install-manager:/$ curl http://github.com/
curl: (6) Could not resolve host: github.com
Well, this seems to continue being an error we're facing so I'll add it as a bug and investigate when I have time. Help is of course welcome as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
I'm not sure if this remains an issue, but I want to keep it open to investigate.
I haven't verified it lately, but it is likely still an issue. I would like to attempt a fix, but haven't had time to start from scratch on the minikube build system.
Yes, looks like it's still an issue.
$ minikube version
minikube version: v1.25.2
commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7-dirty
$ minikube start --kubernetes-version=v1.23.4 --nodes=3 --cni=flannel
$ kubectl get po -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-64897985d-mkch2 1/1 Running 0 74s 10.88.0.2 minikube <none> <none>
kube-system etcd-minikube 1/1 Running 0 82s 192.168.39.41 minikube <none> <none>
kube-system kube-apiserver-minikube 1/1 Running 0 88s 192.168.39.41 minikube <none> <none>
kube-system kube-controller-manager-minikube 1/1 Running 0 82s 192.168.39.41 minikube <none> <none>
kube-system kube-flannel-ds-amd64-5gmcp 1/1 Running 0 74s 192.168.39.41 minikube <none> <none>
kube-system kube-flannel-ds-amd64-cp6ql 1/1 Running 0 13s 192.168.39.66 minikube-m03 <none> <none>
kube-system kube-flannel-ds-amd64-nn5dl 1/1 Running 0 49s 192.168.39.177 minikube-m02 <none> <none>
kube-system kube-proxy-7wlfv 1/1 Running 0 13s 192.168.39.66 minikube-m03 <none> <none>
kube-system kube-proxy-8ncpp 1/1 Running 0 49s 192.168.39.177 minikube-m02 <none> <none>
kube-system kube-proxy-g45dm 1/1 Running 0 74s 192.168.39.41 minikube <none> <none>
kube-system kube-scheduler-minikube 1/1 Running 0 81s 192.168.39.41 minikube <none> <none>
kube-system storage-provisioner 1/1 Running 1 (72s ago) 85s 192.168.39.41 minikube <none> <none>
$ kubectl run nginx --image=nginx
pod/nginx created
$ kubectl exec -ti nginx -- bash
root@nginx:/# curl google.com
curl: (6) Could not resolve host: google.com
$ kubectl delete pod -n kube-system coredns-64897985d-mkch2
pod "coredns-64897985d-mkch2" deleted
[tomas@yggdrasil:~]$ kubectl get po -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default nginx 1/1 Running 0 8m47s 10.244.1.2 minikube-m02 <none> <none>
kube-system coredns-64897985d-p9h7j 1/1 Running 0 89s 10.244.1.3 minikube-m02 <none> <none>
kube-system etcd-minikube 1/1 Running 0 11m 192.168.39.41 minikube <none> <none>
kube-system kube-apiserver-minikube 1/1 Running 0 11m 192.168.39.41 minikube <none> <none>
.
.
.
$ kubectl exec -ti nginx -- bash
root@nginx:/# curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
I just verified the behavior on 1.26.1, and am looking into the cause again to see if this is something I could help fix.
We need another workaround, since moving the CNI config got complicated with Kubernetes 1.24
If it is related to /etc/cni/net.d not being empty, that is ? Suggestion is to move podman and cri-o out...
Basically Kubernetes doesn't work, unless you remove all other software from the (shared) configuration.
That is especially true for flannel, which normally doesn't install on host - but bootstraps from containers
If we cannot reorder it, would it be acceptable to conditionally delete the coredns container as the last part of applying the fabric?
@afbjorklund I might need a little help with this; I'm able to come up with Go code to check for the status.podIP in the coredns pod, but I haven't figured out the right place in the start code to insert this. Each place I try, the coredns pod either does not yet exist, or does not have an IP yet.
Here's an alternate idea, but it would require changes to coredns:
We could propose to add code in coredns's startup that identifies whether it gets an IP from the pod CIDR, and exit after a timeout if its IP doesn't match. To avoid breaking Kubernetes writ large, we could put this functionality behind a feature gate or environment variable that minikube could make use of.
Just confirming this still appears to be broken and noting for folks that run into this that the suggested workaround in the kubevirt docs to delete coredns and disable/enable the kubevirt addon doesn't seem to be working.
Steps to reproduce the issue:
In my case, I am trying to install the kubevirt addon, which creates a pod kubevirt-install-manager in the kube-system namespace, which usually schedules on the second node. This pod attempts to download deployment YAML from the kubevirt project, which fails when DNS is unreachable.
Full output of
minikube logs
command: logs.txtFull output of failed command: