Why minikube uses a separated isolated network on KVM?

[KamranAzeem]

Hi. First, thanks for creating minikube! I have a question, for which I am unable to find a solid answer for last two days. So I thought I would ask here.

To be clear, I am not reporting a bug. This is just a question.

My setup:

[kamran@kworkhorse ~]$ minikube start
😄  minikube v1.9.2 on Fedora 31
    ▪ KUBECONFIG=/home/kamran/.kube/config:/home/kamran/.kube/kubeadm-cluster.conf
✨  Using the kvm2 driver based on existing profile
👍  Starting control plane node m01 in cluster minikube
🔄  Restarting existing kvm2 VM for "minikube" ...
🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
🌟  Enabling addons: dashboard, default-storageclass, helm-tiller, metrics-server, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

❗  /usr/local/bin/kubectl is v1.13.4, which may be incompatible with Kubernetes v1.18.0.
💡  You can also use 'minikube kubectl -- get pods' to invoke a matching version
[kamran@kworkhorse ~]$ 

[kamran@kworkhorse ~]$ minikube status
m01
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

[kamran@kworkhorse ~]$ 

[kamran@kworkhorse docker-to-kubernetes]$ kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE               KERNEL-VERSION   CONTAINER-RUNTIME
minikube   Ready    master   5d12h   v1.18.0   192.168.39.174   <none>        Buildroot 2019.02.10   4.19.107         docker://19.3.8
[kamran@kworkhorse docker-to-kubernetes]$ 

So minikube is working fine.

The simple question is: Why minikube uses an isolated network?

The long question is:

• Why minikube creates an additional isolated network when there is already a "default" network present on the host?
• I see that the minikube VM is connected to two networks. the minikube-net , which is an isolated network, and also the default/virbr0 network, which is a NAT network.
• I see that minikube VM needs to access the internet for pulling necessary images and other stuff - which I makes sense. To get this done, it connects to this additional NAT network. The question here is, why bother creating a separate isolated network, and then wire to another network? If minikube is just connected to one NAT network, it would be possible for it to (a) connect to the internet, and (b) be accessible from the host. The host can access any VMs on the default NAT network, so what is the point of creating an isolated network?
• I mean why bother using two networks, when both objectives can be achieved with just one simple NAT network, which already exists?
• Is there some special idea behind creating a separate isolated network?

I could not understand the reason or wisdom behind it. In past, I have seen somewhat similar behavior with virtualbox too. Since I don't use virtualbox, I really don't remember much on how it works/worked.

I am writing an article on minkube to help the community, and this question is explored in detail in the "advanced" article mentioned below.

These articles are:

• basic,
• advanced

In the "advanced" article, all I could do is assume the reasoning for my observation of the way minikube works. I may be wrong in my assumptions. So before announcing the article to general public, I want to be sure, and I want to fix if I have incorrect understanding about something.

[root@kworkhorse ~]# virsh net-dhcp-leases minikube-net
 Expiry Time           MAC address         Protocol   IP address          Hostname   Client ID or DUID
-----------------------------------------------------------------------------------------------------------
 2020-04-21 00:05:03   30:e0:ec:bf:64:8e   ipv4       192.168.39.174/24   minikube   01:30:e0:ec:bf:64:8e

[root@kworkhorse ~]# 

[root@kworkhorse ~]# virsh net-dhcp-leases default
 Expiry Time           MAC address         Protocol   IP address           Hostname   Client ID or DUID
------------------------------------------------------------------------------------------------------------
 2020-04-21 00:32:39   e8:94:3e:a6:a5:a7   ipv4       192.168.122.134/24   minikube   01:e8:94:3e:a6:a5:a7

[root@kworkhorse ~]# 

I would really appreciate if someone can find some time and shine some light on this. Thank you for your time.

Steps to reproduce the issue: There is no issue. So no steps to reproduce.

[afbjorklund]

The docker-machine VirtualBox network comes with two interfaces, maybe it was just inherited ?

Host cannot access the NAT network, only the Host-only network (which doesn't have internet)

https://www.virtualbox.org/manual/ch06.html

Then SSH is port forwarded on the NAT interface, for management access to the machine:

NIC 1:           MAC: 080027D92509, Attachment: NAT, Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priori
ty: 0, Promisc Policy: deny, Bandwidth group: none
NIC 1 Settings:  MTU: 0, Socket (send: 64, receive: 64), TCP Window (send:64, receive: 64)
NIC 1 Rule(0):   name = ssh, protocol = tcp, host ip = 127.0.0.1, host port = 34522, guest ip = , guest port = 22
NIC 2:           MAC: 0800277F91DC, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported
 speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none

So that it doesn't have to open SSH on the other interface, only the Docker port (2376)

---

There was a backlog item to document this, but it hasn't really been addressed yet either: #4938

SSH listens on all interfaces, but that's another story... For minikube, it also uses port 8443.

[afbjorklund]

It is not unheard of to have two network interfaces in larger Kubernetes deployments, either.

https://www.openshift.com/blog/openshift-container-platform-reference-architecture-implementation-guides

Things will also get more interesting, now that minikube starts supporting multi-node setups...

And the docker driver also changes the rules a bit, since it uses a different CNI network (kindnet).

[KamranAzeem]

Hello. Thanks for the reply. Though I did not get much out of your answer - unfortunately. Here is what I understood from your reply.

• Use of two interfaces/networks was done for VirtualBox some time ago, to cater for some limitations with virtualbox's networking setup. When minikube wanted to extend support to KVM (using the KVM2 driver), somehow that (VirtualBox) setup was inherited. As I see it in reality, there was (is) really no need to have two networks for the minikube VM as the host can access the VM in any NAT network without any problem, and the same VM in the same NAT network can easily connect to the internet. (This (weird networking) has been my one of my reason for my dis-likeness for virtualbox since beginning - which, is a separate story anyway.)
• I am aware that there can be two networks in larger kubernetes networks, or general networks for that matter, however, this is not a use-case (or excuse) for minikube to have two network interfaces. Even if minikube does support multi-node setup in future, it would still not be a use-case, because all the nodes (VMs) of any type of cluster on a virtual NAT network can very easily be accessible from the host, and those VMs can access the internet without any extra configuration. Thus having two separate networks absolutely un-necessary.
• I also think that the purpose of minikube was to allow developers (and sys-admins) easy entry/introduction into kubernetes. If the setup involves multiple networks, it becomes confusing. That is especially true for some developers who do not have understanding about networking, or systems in general.
• I also see that you referenced an issue ( #4938) which you said is related to this issue.

By the way, if you/minikube project wants to use a diagram for minikube setup on Linux/KVM, then you are very welcome to use this one by yours truly: https://github.com/KamranAzeem/docker-to-kubernetes/blob/master/images/minikube-network.png

Thank you for your time.

[KamranAzeem]

Alright, after some investigation, I found some details here: https://github.com/dhiltgen/docker-machine-kvm . I will re-ask this question in that repository.

By the way, I found that minikube's setup does not create the additional host only / isolated network on Hyper-V, the way it did on Linux. Instead it simply uses Hyper-V's default network for accessing the minikube VM from the host, and for the minikube VM to access the Internet. Simple!

I understand that minikube uses docker-machine in the back-ground, and it is their problem (so to speak). However, it would be in public interest to know about these network aspects directly from minikube's documentation, instead of hunting for this information on the internet; because, people are see only "minikube" when they install it! - not docker-machine or boot2docker :)

I have written details about all my findings on the links below, if anyone is interested:

• https://github.com/KamranAzeem/docker-to-kubernetes/blob/master/Minikube-Internals-Fedora.md
• https://github.com/KamranAzeem/docker-to-kubernetes/blob/master/Minikube-Internals-Windows.md

[KamranAzeem]

Alright, after a spiral, following the link https://github.com/kubernetes/minikube/issues/3169, and this https://minikube.sigs.k8s.io/docs/drivers/kvm2/, I reach back here. So, my target audience is probably minikube KVM2 SIG group.

Apparently, now I know some answer to my original question. I would like to propose to please let go of the isolated network thing, as a VM can directly be created on the KVM/libvirt default (NAT) network, and it can be accessed without any problems, and the VM can access the internet. So there is no real use (and benefit) of this isolated network in my opinion. If there is something which I am missing, please enlighten me.

Thanks again. :)

[afbjorklund]

@KamranAzeem : thanks for spending time in investigating this matter!

The details of the docker-machine (and virtualbox driver) and the original kvm driver are now more a historical explanation, rather than a technical motivation. Both projects have stopped updating, and minikube is driving our own versions (of libmachine and kvm2). And like you say, user's dont care.

I suggest opening up a dedicated issue about changing the network layout, and it can be discussed in the light of the different minikube drivers ? As well as making separate Pull Requests for updating the minikube documentation, you can find all of it under the "site" directory in the repository.

The libvirt driver does not have so many users as the others do, so it's a little "behind"

[KamranAzeem]

Hello @afbjorklund , Sorry for the delay in replying. I was doing docker-network-deep-dive :)

Sure, I will open a new/dedicated issue about my suggestion of changing the network layout.

Thanks!

[sharifelgamal]

Has a new issue been open?

[medyagh]

salam @KamranAzeem do you still have this issue, or do you like to elaborate more on this ?

[medyagh]

Hi @KamranAzeem I have not heard from you, I would love to assist wit this issue if you have time and get back to me, if you still have this issue, please re-open

:)