Added hyperkit options for enterprise VPN support

[seborama]

The purpose of these changes is to enhance Hyperkit support from the minikube command line for better integration with enterprise networks behind a VPN.

uuid: Provide VM UUID to restore MAC address (only supported with Hyperkit driver). vpnkitSock: Location of the VPNKit socket used for networking. If empty, disables Hyperkit VPNKitSock, if 'auto' uses Docker for Mac VPNKit connection, otherwise uses the specified VSock." vsockPorts: List of guest VSock ports that should be exposed as sockets on the host (Only supported on with hyperkit now).

[seborama]

Hi,

With regards to testing, below are examples of ~/.minikube/machines/minikube/hyperkit.json with different minikube start options.

EXAMPLE 1

./out/minikube-darwin-amd64 start --vm-driver=hyperkit --cache-images --memory=6144 --logtostderr --loglevel=0 -v=5 --uuid=8de16914-60d3-11e8-b5f4-784f438fc458 --hyperkit-vpnkit-sock=auto --hyperkit-vsock-ports="2376,8443,30000,8080" && ./out/minikube-darwin-amd64 ip

~/.minikube/machines/minikube/hyperkit.json: _Notice the values of the uuid, vsock, vsock_ports, vpnkitsock. Also, the VM was given the same IP address as previously given for this uuid. {"hyperkit":"/usr/local/bin/hyperkit","argv0":"","state_dir":"/Users/someone/.minikube/machines/minikube","vpnkit_sock":"/Users/someone/Library/Containers/com.docker.docker/Data/s50","vpnkit_uuid":"","vpnkit_preferred_ipv4":"","uuid":"8de16914-60d3-11e8-b5f4-784f438fc458","disks":[{"path":"/Users/someone/.minikube/machines/minikube/minikube.rawdisk","size":20000,"format":"","driver":"virtio-blk"}],"iso":["/Users/someone/.minikube/machines/minikube/boot2docker.iso"],"vsock":true,"vsock_ports":[2376,8443,30000,8080],"vsock_guest_cid":3,"vmnet":true,"9p_sockets":null,"kernel":"/Users/someone/.minikube/machines/minikube/bzImage","initrd":"/Users/someone/.minikube/machines/minikube/initrd","bootrom":"","cpus":2,"memory":6144,"console":1,"extra_files":null,"pid":6305,"arguments":["-A","-u","-F","/Users/someone/.minikube/machines/minikube/hyperkit.pid","-c","2","-m","6144M","-s","0:0,hostbridge","-s","31,lpc","-s","1:0,virtio-vpnkit,path=/Users/someone/Library/Containers/com.docker.docker/Data/s50","-s","2:0,virtio-net","-U","8de16914-60d3-11e8-b5f4-784f438fc458","-s","3:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk","-s","4,virtio-sock,guest_cid=3,path=/Users/someone/.minikube/machines/minikube,guest_forwards=2376;8443;30000;8080","-s","5,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso","-s","6,virtio-rnd","-l","com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring","-f","kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"],"cmdline":"/usr/local/bin/hyperkit -A -u -F /Users/someone/.minikube/machines/minikube/hyperkit.pid -c 2 -m 6144M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=/Users/someone/Library/Containers/com.docker.docker/Data/s50 -s 2:0,virtio-net -U 8de16914-60d3-11e8-b5f4-784f438fc458 -s 3:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk -s 4,virtio-sock,guest_cid=3,path=/Users/someone/.minikube/machines/minikube,guest_forwards=2376;8443;30000;8080 -s 5,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso -s 6,virtio-rnd -l com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring -f kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"}

EXAMPLE 2

./out/minikube-darwin-amd64 start --vm-driver=hyperkit --cache-images --memory=6144 --logtostderr --loglevel=0 -v=5 && ./out/minikube-darwin-amd64 ip

~/.minikube/machines/minikube/hyperkit.json: A new IP was provided, no more vsock activation {"hyperkit":"/usr/local/bin/hyperkit","argv0":"","state_dir":"/Users/someone/.minikube/machines/minikube","vpnkit_sock":"","vpnkit_uuid":"","vpnkit_preferred_ipv4":"","uuid":"4b8107f3-656a-11e8-9069-784f438fc457","disks":[{"path":"/Users/someone/.minikube/machines/minikube/minikube.rawdisk","size":20000,"format":"","driver":"virtio-blk"}],"iso":["/Users/someone/.minikube/machines/minikube/boot2docker.iso"],"vsock":false,"vsock_ports":null,"vsock_guest_cid":3,"vmnet":true,"9p_sockets":null,"kernel":"/Users/someone/.minikube/machines/minikube/bzImage","initrd":"/Users/someone/.minikube/machines/minikube/initrd","bootrom":"","cpus":2,"memory":6144,"console":1,"extra_files":null,"pid":8607,"arguments":["-A","-u","-F","/Users/someone/.minikube/machines/minikube/hyperkit.pid","-c","2","-m","6144M","-s","0:0,hostbridge","-s","31,lpc","-s","1:0,virtio-net","-U","4b8107f3-656a-11e8-9069-784f438fc457","-s","2:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk","-s","3,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso","-s","4,virtio-rnd","-l","com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring","-f","kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"],"cmdline":"/usr/local/bin/hyperkit -A -u -F /Users/someone/.minikube/machines/minikube/hyperkit.pid -c 2 -m 6144M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-net -U 4b8107f3-656a-11e8-9069-784f438fc457 -s 2:0,virtio-blk,/Users/someone/.minikube/machines/minikube/minikube.rawdisk -s 3,ahci-cd,/Users/someone/.minikube/machines/minikube/boot2docker.iso -s 4,virtio-rnd -l com1,autopty=/Users/someone/.minikube/machines/minikube/tty,log=/Users/someone/.minikube/machines/minikube/console-ring -f kexec,/Users/someone/.minikube/machines/minikube/bzImage,/Users/someone/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube"}

[seborama]

Sorry, I have found one problem with the parsing of ports. They're defined as int rather than uint16 by Hyperkit and the conversion sometimes fails. Investigating options...

[seborama]

OK, it seems the problem may have been caused by me. I have 3 versions of the driver and 2 of minikube owing to the forked repos and one from homebrew. I've re-sync'ed both minikube and the hyperkit driver from a fresh compilation and the problem has gone away. I can only assume that different versions caused a clash.

[LalatenduMohanty]

@seborama Thanks for the PR, it will be helpful for other projects (including Minishift) using hyperkit.

[seborama]

Hi @LalatenduMohanty / @dlorenc / @praveenkumar , More work would be necessary in a future PR to improve the experience. I'm trying to understand how the minikube VM IP is generated. I seem to gather that the docker hyperkit process requests the IP from the DHCP server based on the generated MAC address of the minikube VM. Is this correct? Could you confirm and enrich my understanding? Several scenarios exist currently which lead to failure when starting minikube as the IP is not generated for the minikube - Hyperkit configuration.

[praveenkumar]

I seem to gather that the docker hyperkit process requests the IP from the DHCP server based on the generated MAC address of the minikube VM. Is this correct?

Yes that's correct https://github.com/machine-drivers/docker-machine-driver-hyperkit/blob/master/pkg/hyperkit/network.go#L48-L65

[praveenkumar]

I can also see this PR is merged from minikube side so @seborama do you think we can merge it here also and then gradually built it by resolving the issues.

[seborama]

Hi Praveen,

Thanks for your reply.

I'd say so yes. The code itself works fine as far as I can tell.

The intricacies (or my understanding thereof 😛) come from how the Minikube VM's IP address is allocated by Hyperkit. Starting a new machine with vsock for the first time won't allocate an IP. The machine needs creating once then deleted and finally re-created with vsock and the uuid of the defunct machine. Awkward. I need to look into the Hyperkit code to gain a better understanding.

[nanzhong]

Is this PR ready to be merged over on this side? I currently make heavy use of this and and building the driver from source, but it would be nice to see these changes merged.

[seborama]

It is for this iteration. I think that further enhancements can be part of separate PR's. For instance, we found that routing in the Minikube K8s cluster to external addresses is not as functional as with say VirtualBox for instance.

[seborama]

Hi @praveenkumar,

Is there anything outstanding on this PR?

Cheers

[seborama]

Hey @nanzhong ,

Currently the code from this PR exists both in the minikube repo (already merged by @dlorenc ) and this repo (this PR, pending).

This means that you can compile the drivers from the Minikube repo (use make driver and optionally make install install-hyperkit-driver or copy the file file manually from the out directory and apply the correct permissions).

HOWEVER... As documented at https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperkit-driver, I've had a look at the binary distribution at https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit.

The binary has been updated from the minikube repo. So you don't need to compile it yourself if you don't want to, just follow the steps in the doc from the minikube project (above link).

Best of luck.

PS: details of the driver update as reported at https://storage.googleapis.com/minikube/: (in short: yesterday)

<Contents>
<Key>releases/latest/docker-machine-driver-hyperkit</Key>
<Generation>1532362270225121</Generation>
<MetaGeneration>1</MetaGeneration>
<LastModified>2018-07-23T16:11:10.224Z</LastModified>
<ETag>"6c90be83e9882c5bd838e2bb8461dc59"</ETag>
<Size>26811748</Size>
</Contents>

[nanzhong]

A perfect, thanks!

[seborama]

Hi, This PR is stale. I'm closing it. Best regards.