Switch to the rkt iso by default

[dlorenc]

Ref #168 #570

[philips]

@dlorenc is the plan to move the code into this tree?

[dlorenc]

Yeah, if @s-urbaniak agrees.

[s-urbaniak]

@dlorenc @philips sure thing, I agree, we should fix the OSX issues first though. I'll try to grab a Macbook during the weekend (Linux-only household here).

[ryanj]

@s-urbaniak have info on how to reproduce the OSX issues?
I was able to run a quick test of your v0.0.3 image using minikube 0.10. I didn't notice any major issues while testing.

[philips]

@ryanj on virtualbox or vmware fusion?

[philips]

OK, so a few things:

1. Something about minikube causes vmwarefusion to crash if the box doesn't come up perfectly. Doing open $HOME/.minikube/machines/minikube/minikube.vmx returns an error. Reboot required to fix.
2. Unpacking initramfs took 100 seconds (screenshot below)
3. The machine doesn't seem to have a network interface

@s-urbaniak so, I think you need to add some network drivers first to get this working

[philips]

@s-urbaniak I need vmxnet3 please as a kernel driver!

[philips]

[s-urbaniak]

@philips I added the driver in the minikube iso. Do you mind trying it out with https://github.com/coreos/minikube-iso/releases/download/v0.0.4/minikube-v0.0.4.iso?

[aecolley]

On OS X, using the new rkt v0.0.3 iso, docker run -v mounts no longer work. That is, they no longer mount directories from the OS X host into the docker container. The default iso manages this sorcery OK, and it would be nice if it was supported in the rkt iso too.

[philips]

@aecolley can you give us an example?

[ryanj]

the iso booted cleanly for me using Virtualbox 5.1 on OSX 10.10.2 and Virtualbox 5.0 on Fedora24. I didn't test the other providers, thanks for the additional info

I successfully deployed an image from dockerhub (didn't test volume mounts), and was able to view the running container in the output of rkt list after connecting to the node via minikube ssh.

[philips]

@aecolley can you give details on your OS, hypervisor, and version of minikube along with the example?

[aecolley]

@philips Certainly. Here's my test case script:

#!/bin/bash
minikube stop || :
minikube delete || :
rm -fr mountable || :

set -o errexit
minikube start "$@"
dvm use 1.12.1
eval "$(minikube docker-env)"
mkdir mountable
touch mountable/proof
output="$(docker run --rm -v $PWD/mountable:/mounted:ro busybox:latest /bin/ls -asCF /mounted)"
printf 'Output: %s\n' "${output}"
if [[ "${output}" == *proof* ]]; then
    echo PASS
    exit 0
else
    echo FAIL
    exit 1
fi

And here's an unedited copy-paste of the Terminal session in which I show it failing with the rkt iso and passing with the default setup. At the end, it shows the OS/virtualbox/minikube/docker versions as requested.

MacBook-Pro-2:~ aecolley$ tmp/testcase.sh --network-plugin=cni --container-runtime=rkt --iso-url=https://github.com/coreos/minikube-iso/releases/download/v0.0.3/minikube-v0.0.3.iso
Stopping local Kubernetes cluster...
Machine stopped.
Deleting local Kubernetes cluster...
Machine deleted.
Starting local Kubernetes cluster...
Kubectl is now configured to use the cluster.
Now using Docker 1.12.1
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
8ddc19f16526: Pulling fs layer
8ddc19f16526: Verifying Checksum
8ddc19f16526: Download complete
8ddc19f16526: Pull complete
Digest: sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6
Status: Downloaded newer image for busybox:latest
Output: total 4
     0 ./     4 ../
FAIL
MacBook-Pro-2:~ aecolley$ tmp/testcase.sh
Stopping local Kubernetes cluster...
Machine stopped.
Deleting local Kubernetes cluster...
Machine deleted.
Starting local Kubernetes cluster...
Kubectl is now configured to use the cluster.
Now using Docker 1.12.1
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
8ddc19f16526: Pulling fs layer
8ddc19f16526: Verifying Checksum
8ddc19f16526: Download complete
8ddc19f16526: Pull complete
Digest: sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6
Status: Downloaded newer image for busybox:latest
Output: total 4
     0 ./        4 ../       0 proof
PASS
MacBook-Pro-2:~ aecolley$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1004
MacBook-Pro-2:~ aecolley$ docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 31
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Security Options:
Kernel Version: 4.4.14-boot2docker
Operating System: Boot2Docker 1.11.1 (TCL 7.1); master : 901340f - Fri Jul  1 22:52:19 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 995.9 MiB
Name: minikube
ID: OXA2:LCBA:ACGN:HZZU:AUKE:YX3U:QCXI:B4WI:CLQH:5PC3:RFQT:KWNT
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 28
 Goroutines: 57
 System Time: 2016-09-17T22:16:05.318013662Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8
MacBook-Pro-2:~ aecolley$ minikube version
minikube version: v0.10.0
MacBook-Pro-2:~ aecolley$ VBoxManage --version
5.1.6r110634

The version numbers are the same in both cases, but the other parts of the docker info output is different. For completeness, here's the output of docker info with the rkt iso loaded:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.11.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Security Options:
Kernel Version: 4.7.2
Operating System: Buildroot 2016.08-rc3
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 993.4 MiB
Name: minikube
ID: U3OT:RIAO:ZX5D:RKBD:HC3J:O5TF:WNRA:CCKH:P7NK:FBQW:IFBI:P5R5
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Insecure Registries:
 127.0.0.0/8

Finally, it also fails when using the docker runtime with the rkt iso (i.e. with testcase.sh --container-runtime=docker --iso-url=https://github.com/coreos/minikube-iso/releases/download/v0.0.3/minikube-v0.0.3.iso), so I believe it's a problem with the iso and not with the rkt support.

[philips]

@s-urbaniak put up a new version of the ISO which WFM! https://github.com/coreos/minikube-iso/releases/download/v0.0.4/minikube-v0.0.4.iso Confirmed working with minikube v0.10.0 and OSX and vmware fusion. Thanks @s-urbaniak

[philips]

@s-urbaniak what is the next step to move to this ISO by default?

[s-urbaniak]

@philips I can prepare a PR against minikube in the next week, I am still working on xhyve support though: https://github.com/coreos/minikube-iso/issues/17

@dlorenc Do you see xhyve support as a blocker right now or do you think I can proceed with a "boot2docker to minikube-iso" PR?

[philips]

I personally don't see xhyve as a blocker as it is still really hard to get that working. We need to figure out how to bundle it instead of relying on the libmachine external driver, etc.

[dlorenc]

I suppose we could make this iso default automatically for people passing the container-runtime=rkt flag, but there are quite a few xhyve users that I'd rather not break.

I think the xhyve issues are fixed now though, I'll do some final testing tomorrow.

@s-urbaniak do you want to send a PR to start moving the ISO build scripts from your repo here?

[philips]

@dlorenc @s-urbaniak mentioned he got xhyve working yesterday after I commented.

[s-urbaniak]

@dlorenc working on the changes to pull the ISO build scripts here right now. Not sure if I'll make it today though.

xhyve basically works according to https://github.com/coreos/minikube-iso/issues/17#issuecomment-250664776, @zchee is trying to push this upstream via https://github.com/docker/hyperkit/issues/63.

[zchee]

@dlorenc @philips @s-urbaniak As in the https://github.com/docker/hyperkit/issues/63, I still haven't understood the method of calculating the ramdisk_start value. But I was some debugging, It seems to be able to boot any ISO if about twice to ramdisk_start. (well, it also would be incorrect)

If there is no response from docker-dev until merge https://github.com/zchee/docker-machine-driver-xhyve/pull/138( waiting for @dlorenc check done), I was thinking to temporary fix of kexec.c for rkt.

FYI, I'm already using the zchee/libhyperkit that the static vendoring hyperkit and other patched(cgo problem, dispatch_loop(Apple GCD), lib9p server), so we do not need to wait for issue response. We can support rkt immediately.

How you do you feel about it?

[s-urbaniak]

@zchee I'd say go with it :-)

[dlorenc]

+1, that sounds great to me.

[zchee]

@s-urbaniak @dlorenc @philips Done :) Please check and test for it. https://github.com/zchee/docker-machine-driver-xhyve/pull/140

I was tested and successful with

minikube start --show-libmachine-logs=true \
               --v=7 \
               --logtostderr=true \
               --vm-driver="xhyve" \
               --memory=4096 \
               --cpus=2 \
               --network-plugin=cni \
               --container-runtime=rkt \
               --iso-url=https://github.com/coreos/minikube-iso/releases/download/xhyve/minikube-xhyve.iso

If you want to pre-compile, Here is lib9p qcow2 static linked binary. build by my local env. http://get.zchee.io/bin/docker-machine-driver-xhyve

[s-urbaniak]

I suggest to reopen this, as we have the scripts now available to generate the ISO, but inside minikube we are still pointing to the boot2docker based iso image as per [1].

@dlorenc @r2d4 I don't see pointers in the source on how the upload is happening on https://storage.googleapis.com/minikube. Please let me know if there is anything on how I can help to make this happen, so we can shut down https://github.com/coreos/minikube-iso/releases :-)

[1] https://github.com/kubernetes/minikube/blob/3a61ad6ab2a752628385386505513ef12edd94b6/pkg/minikube/constants/constants.go#L62

[r2d4]

I reopened this, I think it got auto-closed from the PR. Right now, we are just pushing the iso with gsutil https://github.com/kubernetes/minikube/tree/master/deploy/iso/boot2docker

I can push the previous releases of coreos/minikube-iso to our storage bucket. I'll let you know when its done.

Another thing we might need to figure are names for the ISOs - the b2d one is currently called minikube.iso and this one is minikube-iso.iso. Until we make this one the default, we should probably rename it something else, possibly minikube-alpha.iso? Any suggestions @dlorenc or @s-urbaniak?

[s-urbaniak]

minikube-alpha.iso sgtm for the time being.

[dlorenc]

I think we're basically ready to make this default. Any concerns about doing this in 0.13?