alternative boot2docker ISO image (minikube-iso) support

[s-urbaniak]

Hi,

I am building an alternative boot2docker image called minikube-iso https://github.com/coreos/minikube-iso for the https://github.com/kubernetes/minikube project which include systemd, as well as the rkt container runtime. The ISO is built using http://buildroot.org.

When starting using the following command:

$ minikube start --show-libmachine-logs=true --v=7 --logtostderr=true --iso-url=https://github.com/coreos/minikube-iso/releases/download/xhyve/minikube-xhyve.iso --vm-driver=xhyve
I0920 13:37:51.313538   78441 notify.go:103] Checking for updates...
Starting local Kubernetes cluster...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:65105
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Creating CA: /Users/sur/.minikube/certs/ca.pem
Creating client certificate: /Users/sur/.minikube/certs/cert.pem
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.2.3 (Homebrew) =====
(minikube) DBG | 
(minikube) DBG | executing: /usr/local/bin/VBoxManage -v
(minikube) DBG | STDOUT: 5.0.22r108108
(minikube) DBG | 
(minikube) DBG | STDERR: 
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/sur/.minikube/cache/boot2docker.iso from file:///Users/sur/.minikube/cache/iso/minikube-xhyve.iso...
(minikube) Creating VM...
(minikube) DBG | Mounting boot2docker.iso
(minikube) Extracting vmlinuz64 and initrd.img from boot2docker.iso...
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil attach /Users/sur/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/sur/.minikube/machines
/minikube/b2d-image] []  <nil> 0xc820026010 0xc820026018 [] <nil> <nil> <nil> <nil> false [] [] [] [] <nil>} attach /Users/sur/.minikube/machines/minikube/boot
2docker.iso -mountpoint /Users/sur/.minikube/machines/minikube/b2d-image
(minikube) /dev/disk9                                           /Users/sur/.minikube/machines/minikube/b2d-image
(minikube) DBG | Extracting vmlinuz64 into /Users/sur/.minikube/machines/minikube
(minikube) DBG | Extracting initrd.img into /Users/sur/.minikube/machines/minikube
(minikube) DBG | Unmounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil detach /Users/sur/.minikube/machines/minikube/b2d-image] []  <nil> 0xc820026010 0xc820026018 [] <nil> <
nil> <nil> <nil> false [] [] [] [] <nil>} detach /Users/sur/.minikube/machines/minikube/b2d-image
(minikube) "disk9" unmounted.
(minikube) "disk9" ejected.
(minikube) Generating 20000MB disk image...
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil create -megabytes 20000 -type SPARSEBUNDLE /Users/sur/.minikube/machines/minikube/root-volume] []  <nil
> 0xc820026010 0xc820026018 [] <nil> <nil> <nil> <nil> false [] [] [] [] <nil>} create -megabytes 20000 -type SPARSEBUNDLE /Users/sur/.minikube/machines/miniku
be/root-volume
(minikube) created: /Users/sur/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) Creating SSH key...
(minikube) Fix file permission...
(minikube) DBG | /Users/sur/.minikube/machines/minikube/boot2docker.iso
(minikube) DBG | /Users/sur/.minikube/machines/minikube/config.json
(minikube) DBG | /Users/sur/.minikube/machines/minikube/id_rsa
(minikube) DBG | /Users/sur/.minikube/machines/minikube/id_rsa.pub
(minikube) DBG | /Users/sur/.minikube/machines/minikube/initrd.img
(minikube) DBG | /Users/sur/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) DBG | /Users/sur/.minikube/machines/minikube/vmlinuz64
(minikube) Generate UUID...
(minikube) DBG | Generated UUID: 3FDDB05E-2DC8-40BB-AB35-F577EA3A43D2
(minikube) Convert UUID to MAC address...
(minikube) DBG | Converted MAC address: 7a:24:5a:36:cf:32
(minikube) Starting minikube...
(minikube) DBG | [xhyve -A -U 3FDDB05E-2DC8-40BB-AB35-F577EA3A43D2 -c 1 -m 1024M -l com1,autopty -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -s 3,ahci-cd,/Us
ers/sur/.minikube/machines/minikube/boot2docker.iso -s 4:0,ahci-hd,/dev/rdisk9 -f kexec,/Users/sur/.minikube/machines/minikube/vmlinuz64,/Users/sur/.minikube/m
achines/minikube/initrd.img,loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 base host=minikube -F /Users/sur/.minikube
/machines/minikube/minikube.pid -s 5,virtio-9p,host=/Users]
(minikube) Waiting for VM to come online...
(minikube) DBG | 
(minikube) DBG | Not there yet 1/60, error: IP not found for MAC 7a:24:5a:36:cf:32 in DHCP leases
(minikube) DBG | operation not supported by device
(minikube) Waiting on a pseudo-terminal to be ready... done
(minikube) Hook up your terminal emulator to /dev/ttys005 in order to connect to your VM
(minikube) DBG | vm exit[0]
(minikube) DBG |        reason          VMX
(minikube) Hook up a terminal emulator to /dev/ttys005 in order to access your VM
(minikube) DBG |        rip             0xffffffff81908cdd
(minikube) DBG |        inst_length     5
(minikube) DBG |        status          0
(minikube) DBG |        exit_reason     2
(minikube) DBG |        qualification   0x0000000000000000
(minikube) DBG |        inst_type               0
(minikube) DBG |        inst_error              0
(minikube) DBG | signal: abort trap &{0xc82000e140} &{0xc82000e160}
(minikube) DBG | 
(minikube) DBG | Not there yet 2/60, error: IP not found for MAC 7a:24:5a:36:cf:32 in DHCP leases
(minikube) DBG | 
(minikube) DBG | Not there yet 3/60, error: IP not found for MAC 7a:24:5a:36:cf:32 in DHCP leases
...

Especially the vm exit[0] apparently indicates something, but I am not able to reason about the output. I am sorry, that this lies outside of the "regular" boot2docker ISO, but any help/hint would be helpful. Am i missing some kernel module maybe?

Here is the kernel configuration: https://github.com/s-urbaniak/minikube-iso/blob/xhyve-support/board/coreos/minikube/linux-4.7_defconfig

PS: Creating a minikube instance using minikube start --vm-driver=xhyve works.

Thanks for any hints!

[zchee]

@s-urbaniak Hmm... ok. I'm do not yet understand that's error meaning, but I remember seeing (recognize? look familiar? sorry I'm not good at English ) that error. Please wait a moment...

(If I remember right, this error also occurs fuchsia-mirror/magenta's forked qemu. I will investigate.)

[zchee]

@s-urbaniak Ah, remembered. fuchsia-mirror/magenta also uses buildroot kernel. If you have time, try run the other tiny OS? (but, I will continue the investigate)

[zchee]

@s-urbaniak Also, If you have time, could you try launch that ISO uses original xhyve or hyperkit?

If both of failed, docker-machine-driver-xhyve also not supported. Needs C languages level hack(or workaround?).

Edit: If you not familiar with xhyve or hyperkit, I will test it.

[s-urbaniak]

@zchee not too familiar to be honest, since I am actually a Linux user only ;-) If you have spare cycles to give it a shot with xhyve, that would be great :-)

[zchee]

@s-urbaniak OK, np! So that question for minikube/minishift or other kubenetes?

Anyway, I will investigate later :)

[s-urbaniak]

@zchee this is for minikube :-)

[zchee]

@s-urbaniak ok :)

[zchee]

@s-urbaniak Sorry, reference for me. What kubenetes team's goal? Use rkt?

[s-urbaniak]

@zchee Kubernetes as of 1.3 also supports rkt as an additional container runtime next to Docker. Furthermore there is a container runtime interface specification out there [1] which abstracts container runtime support inside Kubernetes even more.

[1] https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/container-runtime-interface-v1.md

[zchee]

@s-urbaniak Ah, sorry. My question is bad... :(

In other words, What I should be run any iso? If want to run the buildroot iso, ok, I will debug it use buildroot iso. But If you want to run other iso, It might be wasted.

So, What do you want to run the iso file instead of boot2docker.iso?

I cannot use English well, so please forgive me if I say something rude. Edit: or, sorry if I'm misunderstood.

[s-urbaniak]

@zchee no, problem, I appreciate the help :-)

To try out the ISO, simply execute:

$ minikube start \
  --show-libmachine-logs=true --v=7 --logtostderr=true \
  --iso-url=https://github.com/coreos/minikube-iso/releases/download/xhyve/minikube-xhyve.iso \
  --vm-driver=xhyve

Note: you must have minikube 0.10.0 (https://github.com/kubernetes/minikube/releases/tag/v0.10.0).

[zchee]

@s-urbaniak OK thanks. will try it.

BTW, I've also joined kubenetes Slack channel. Could you help in there? If ok, #minikube channel?

[s-urbaniak]

@zchee sure thing!

[zchee]

Current state: https://github.com/coreos/minikube-iso/issues/17

[dlorenc]

So with the two PRs I just sent: #144 and #143 I got this to boot with the new ISO!

$ minikube start --iso-url=http://storage.googleapis.com/minikube/iso/buildroot/minikube-v0.0.6.iso --vm-driver=xhyve --show-libmachine-logs --v=100

Unfortunately this doesn't work with the old ISO though. I took a look and the xhyve commands run seem the same, but I might be missing something.

@zchee is there anything else missing in this branch to work with both ISOs?

[dlorenc]

Nevermind, I'm not sure what happened but my PR #144 works with both ISOs :)

[zchee]

xhyve driver already support minikube iso. But related issue now discuss on https://github.com/zchee/docker-machine-driver-xhyve/pull/147. I'll close this issue thread after the finished that discuss.

[zchee]

147 has been merged, and I was released v0.3.0.

Close.

[psinha260689]

in boot2doecker when i start minikube , they ask minikube login and minikube password. what are the default values ?