search

moby / hyperkit

A toolkit for embedding hypervisor capabilities in your application
BSD 2-Clause "Simplified" License
3.61k stars 327 forks source link

Allow the VM to be a router #178

Open burdiyan opened 6 years ago

burdiyan commented 6 years ago

First of all, it worth mentioning that I'm not a networking expert by any means and not even sure that this is right repo to file this issue, but I've made quite some tests with all that.

The problem is that seems like Hyperkit VM is not able to work as a router, when routing of the host os is configured to do so.

Use Case

Briefly, the use case is to be able to access containers running inside the Hyperkit virtual machine by IP and name from the host machine (Mac OS). It doesn't matter whether these are docker container or Kubernetes pods.

The use case for that is described in these issues in more details:

https://github.com/moby/hyperkit/issues/55 https://github.com/docker/for-mac/issues/155

I'm not sure why it's working for VirtualBox and not for Hyperkit.

Problem

Basically I was spinning up virtual machines based on boot2docker.iso using the same ISO I can't route traffic through the VM with Hyperkit, but a do can with VirtualBox.

Example using https://github.com/kubernetes/minikube on MacOS High Sierra:

  1. minikube start --vm-driver hyperkit.
  2. kubectl run nginx --image nginx:alpine.
  3. kubectl get pods -o wide (note the IP of the pod and find out it's CIDR, let's say it's 172.18.0.0/16).
  4. sudo route -n add 172.18.0.0/16 $(minikube ip).
  5. Use curl to access the IP of the nginx pod. In my case it hangs until timeout.

Doing the same but with VirtualBox driver works totally fine.

It seems like there is something inside Hyperkit (or maybe underlying MacOS hypervisor that does not let packets going through).

Although doing tcpdump on the Minikube interface (normally named bridge0) show that packets are routed correctly.

rn commented 6 years ago

I believe this is/might be possible if you use the vmnet networking backend, however this requires hyperkit to be run as root. How does minikube start hyperkit? This may be a better issue on the minikube repositories...

justincormack commented 6 years ago

minikube says "The hyperkit driver currently requires running as root to use the vmnet framework to setup networking." so it sounds like it is doing the right thing. I have had this work in the past using the vmnet driver. I don't think it not working is a hyperkit issue though; we do not have any control over routing for the vmnet case, as it is all handled by the OS.