Support the rkt/minikube-iso

zchee commented 7 years ago

Support the rkt/minikube-iso.

Remove go generate mv .o .syso, use #cgo LDFLAGS: -lfoo.o
Temporary fix ramdisk_start to twice size.
- It would be incorrect but can boot large size kernel.
- Tested tinycore, debian testing, ubuntu yakkety, minikube-iso and boot2docker
Support any filename of kernel executable and initrd image.
- Because of different filename by distro, tools or etc. such as build by Buildroot.

zchee commented 7 years ago

/cc @dlorenc @r2d4 Please test it use minikube with https://github.com/coreos/minikube-iso/releases/download/xhyve/minikube-xhyve.iso minikube-v0.0.5.iso is bzImage and initrd so can not yet boot. If try build, GO_BUILD_TAGS='lib9p qcow2' make V=1. not "GO_BUILD_TAG", add S. sorry.

/cc @s-urbaniak Added d.Vmlinuz and d.Initrd. They are initialized d.SetConfigFromFlags if use docker-machine. Assumed boot2docker.iso if empty. According to some document, /boot directory is a specification. So hardcode at https://github.com/zchee/docker-machine-driver-xhyve/blob/minikube-iso_kexec/xhyve/xhyve.go#L606-L607

Is there no problem this fix?

zchee commented 7 years ago

I will release 0.3.0 version after the merge it and some fix.

s-urbaniak commented 7 years ago

@zchee LGTM! I'll prepare a PR against minikube to configure the driver correctly with the minikube-iso image.

zchee commented 7 years ago

@s-urbaniak Thanks for test :) Did you say PR means for xhyve-driver? minikube? If minikube, I was confirmed successful with this branch, just for reference. https://github.com/zchee/minikube/commit/b6461637f9a976477b99d59317bda0b3bc21c77a

s-urbaniak commented 7 years ago

@zchee I would prepare a PR against minikube like you did in https://github.com/zchee/minikube/commit/b6461637f9a976477b99d59317bda0b3bc21c77a :-) Although looking at your change I am wondering how to "autodetect" whether it is a minikube-iso or boot2docker, maybe by looking at the --iso-url string.

zchee commented 7 years ago

@s-urbaniak Ah, yeah. It supports minikube-iso only(Sorry, it's my quick hack for minikube-iso test lol), So if set --iso-url with boot2docker.iso from minikube side, will fail.

It would be better to implement the processing. Which is implement parse of iso? xhyve side? minikube?

Edit: Do you think which one is better?

s-urbaniak commented 7 years ago

Hmm ... I just tried out this PR, and I get this error:

$ minikube --vm-driver=xhyve start --show-libmachine-logs --v=5 --alsologtostderr --stderrthreshold 5
W1012 14:21:13.055893    4662 root.go:127] Error reading config file at /Users/sur/.minikube/config/config.json: open /Users/sur/.minikube/config/config.json: no such file or directory
I1012 14:21:13.056541    4662 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
Found binary path at /Users/sur/src/go/src/github.com/zchee/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:49533
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Creating CA: /Users/sur/.minikube/certs/ca.pem
Creating client certificate: /Users/sur/.minikube/certs/cert.pem
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.2.3 (7d55ee7) =====
(minikube) DBG | 
(minikube) DBG | executing: /usr/local/bin/VBoxManage -v
(minikube) DBG | STDOUT: 5.1.6r110634
(minikube) DBG | 
(minikube) DBG | STDERR: 
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/sur/.minikube/cache/boot2docker.iso from file:///Users/sur/.minikube/cache/iso/minikube-0.7.iso...
(minikube) Creating VM...
(minikube) Extracting kernel and initrd from boot2docker.iso...
(minikube) DBG | Mounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil attach /Users/sur/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/sur/.minikube/machines/minikube/b2d-image] []  <nil> 0xc420086008 0xc420086010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} attach /Users/sur/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/sur/.minikube/machines/minikube/b2d-image
(minikube) /dev/disk5                                           /Users/sur/.minikube/machines/minikube/b2d-image
(minikube) DBG | Extracting kernel into /Users/sur/.minikube/machines/minikube
E1012 14:21:23.427927    4662 start.go:87] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: open /Users/sur/.minikube/machines/minikube: is a directory. Retrying.

PS: The master branch works.

zchee commented 7 years ago

@s-urbaniak Ah, really...? work fine this pull request :( but I was fixed several times this pull request, and added some better log messages. Could you build again?

Edit: pre-compile binary and test iso

http://get.zchee.io/bin/docker-machine-driver-xhyve

shasum -a 256 ./bin/docker-machine-driver-xhyve
44cb25be703e416202ed24189c4b26fbc37342de489bbf53fc99140ae9b14acc  ./bin/docker-machine-driver-xhyve

http://get.zchee.io/bin/minikube

https://github.com/zchee/minikube/commit/b6461637f9a976477b99d59317bda0b3bc21c77a

shasum -a 256 minikube
869880dbc61074fc49d018549274e877cb117698598f673338e5d0e9a32d5114  minikube

http://s3.zchee.io/iso/minikube-virtblk.iso

https://github.com/coreos/minikube-iso/commit/73ccecaeff4493a4d52a1041b82b8440c1c2ffca

shasum -a 256 minikube-virtblk.iso
93301b7231f94bc4b1662ac486dd09e01c3b34b45947346f759370300861aeef  minikube-virtblk.iso

zchee commented 7 years ago

/cc @dlorenc @r2d4 @s-urbaniak Added latest docker/hyperkit including fixed kexec.c. PTAL.

r2d4 commented 7 years ago

@zchee I had some trouble creating the vm. I'm get the same error as s-urbaniak above

E1018 19:57:31.013713 1536 start.go:93] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: open /Users/mrick/.minikube/machines/minikube: is a directory

zchee commented 7 years ago

@r2d4 Hmm... is this binary build from source with this PR? I'll try test later.

zchee commented 7 years ago

@s-urbaniak @r2d4 Ah, repro on latest minikube ~~I'll fix it. sorry.~~

Edit: In @r2d4 case, maybe needs https://github.com/zchee/minikube/commit/b6461637f9a976477b99d59317bda0b3bc21c77a In @s-urbaniak case, maybe that is old binary I think.

zchee commented 7 years ago

@dlorenc @r2d4 @s-urbaniak I can not release the new version because waiting for merge this pull request. How is the progress on minikube and minikube-iso?

s-urbaniak commented 7 years ago

@zchee since I wasn't able to get the xhyve driver running locally with this PR, I was not able to progress on the minikube side. When I'll be back in the office next week, I can retry using this PR, and the latest minikube on master.

dlorenc commented 7 years ago

Hey,

I'm trying to build this locally and I'm getting this error:

duplicate symbol _caml_tuplify5 in:
    mirage_block_ocaml.syso
    vendor/github.com/zchee/libhyperkit/mirage_block_ocaml.o
duplicate symbol _caml_tuplify7 in:
    mirage_block_ocaml.syso
    vendor/github.com/zchee/libhyperkit/mirage_block_ocaml.o
duplicate symbol _caml_tuplify8 in:
    mirage_block_ocaml.syso
    vendor/github.com/zchee/libhyperkit/mirage_block_ocaml.o
ld: 14678 duplicate symbols for architecture x86_64

Any ideas?

zchee commented 7 years ago

@dlorenc It seems to exists mirage_block_ocaml.o and mirage_block_ocaml.syso file on any place. maybe on the vendor/github.com/zchee/libhyperkit. Please remove mirage_block_ocaml.syso, or just make clean before building. https://github.com/zchee/docker-machine-driver-xhyve/blob/6e75aa47e6e89aae20a83de923019dc774f6f5a2/Makefile#L219-L220

dlorenc commented 7 years ago

Ah nice, make clean wasn't working but I recloned the repository and got it to build.

$ minikube version
minikube version: v0.12.1
$ GO_BUILD_TAGS='lib9p qcow2' make V=1 install

but I'm now getting the same error as @r2d4.

dlorenc commented 7 years ago

I think I found the issue. I'll send a few PRs in a bit.

zchee commented 7 years ago

@dlorenc Ah, I was talking on the slack with @r2d4, This pull request needs this change on minikube side. https://github.com/zchee/minikube/commit/b6461637f9a976477b99d59317bda0b3bc21c77a Because I added initrd and vmlinuz struct variable.

If call xhyve-driver by minikube, will initrd and vmlinuz is empty. So occur error.

However, this problem might be solved with adding initialize method. It means like

Sorry, I still don't understand minikube internal. Where do I should add the initialize method for having a backward compatibility of minikube?

dlorenc commented 7 years ago

Where do I should add the initialize method for having a backward compatibility of minikube?

Maybe in here: https://github.com/kubernetes/minikube/blob/master/pkg/minikube/cluster/cluster_darwin.go#L63

?

zchee commented 7 years ago

@dlorenc Ah, sorry. Maybe I misunderstood and mistaken... Please give me some time, I'll fix this pull request.

dlorenc commented 7 years ago

Sure! Let me know if I can help.

zchee commented 7 years ago

@dlorenc Thanks.

BTW, I want to local test use original minikube-alpha(?) iso file. That's has been released? or When scheduled for release?

dlorenc commented 7 years ago

Not sure what you mean. You can grab the rkt/systemd ISO at the URL in the doc here: https://github.com/kubernetes/minikube/blob/master/README.md#using-rkt-container-engine

zchee commented 7 years ago

@dlorenc It means http://storage.googleapis.com/minikube/iso/buildroot/minikube-v0.0.5.iso ? I tested, but that has not CONFIG_VIRTIO_BLK=y. https://github.com/coreos/minikube-iso/pull/28

In other words, When you plan to minikube support of qcow2 disk image?

s-urbaniak commented 7 years ago

Indeed the virtio config is not present in the 0.0.5 release yet, as coreos/minikube-iso#28 was merged after the release. dlorenc let me know if I can help out with releasing a 0.0.6, which would include the above kernel config.

dlorenc commented 7 years ago

@r2d4 just released 0.6 yesterday.

zchee commented 7 years ago

@dlorenc thanks for release 0.6.

zchee commented 7 years ago

@dlorenc Merged your two PR. thanks. Ready to merge it?

dlorenc commented 7 years ago

Yeah, doing some final sanity testing now :)

@r2d4 could you try this out today too?

zchee commented 7 years ago

@dlorenc Got it. I'll re-test with docker-machine and minikube.

dlorenc commented 7 years ago

Nice! Everything works for me. Merge whenever :)

zchee commented 7 years ago

@dlorenc

@r2d4 could you try this out today too?

Do I need to wait?

dlorenc commented 7 years ago

Nah, it should be safe to merge now.

zchee commented 7 years ago

Merged. Thanks!

r2d4 commented 7 years ago

It works on my machine also 👍

zchee commented 7 years ago

@dlorenc @r2d4 @s-urbaniak I'll release next version after some fix (e.g. add ocaml setup to travis.yml) Thanks for a lot of review and test.

s-urbaniak commented 7 years ago

@zchee :+1: thanks a lot!!!

machine-drivers / docker-machine-driver-xhyve

Support the rkt/minikube-iso #140