kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.41k stars 4.88k forks source link

macOS: Can't create minikube with xhyve #708

Closed pquentin closed 7 years ago

pquentin commented 8 years ago

Minikube version (use minikube version): 0.11.0

Environment:

What happened:

My team uses Kubernetes in production and we recently decided to switch local development from docker-compose to minikube. Their migration went well, but not mine. I tried to create a minikube cluster like this:

$ docker -v
Docker version 1.12.2, build bb80604
$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.
$ minikube start --vm-driver=xhyve --show-libmachine-logs --v=10 --alsologtostderr
I1018 14:54:32.935093   73832 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:61922
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.2.3 (Homebrew) =====
(minikube) DBG |
(minikube) DBG | executing: /usr/local/bin/VBoxManage -v
(minikube) DBG | STDOUT: 5.0.4r102546
(minikube) DBG |
(minikube) DBG | STDERR:
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/quentin/.minikube/cache/boot2docker.iso from file:///Users/quentin/.minikube/cache/iso/minikube-0.7.iso...
(minikube) Creating VM...
(minikube) Extracting vmlinuz64 and initrd.img from boot2docker.iso...
(minikube) DBG | Mounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil attach /Users/quentin/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/quentin/.minikube/machines/minikube/b2d-image] []  <nil> 0xc42002e010 0xc42002e018 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} attach /Users/quentin/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) /dev/disk4                                           /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) DBG | Extracting vmlinuz64 into /Users/quentin/.minikube/machines/minikube
(minikube) DBG | Extracting initrd.img into /Users/quentin/.minikube/machines/minikube
(minikube) DBG | Unmounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil detach /Users/quentin/.minikube/machines/minikube/b2d-image] []  <nil> 0xc42002e010 0xc42002e018 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} detach /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) "disk4" unmounted.
(minikube) "disk4" ejected.
(minikube) Generating 20000MB disk image...
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil create -megabytes 20000 -type SPARSEBUNDLE /Users/quentin/.minikube/machines/minikube/root-volume] []  <nil> 0xc42002e010 0xc42002e018 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} create -megabytes 20000 -type SPARSEBUNDLE /Users/quentin/.minikube/machines/minikube/root-volume
(minikube) created: /Users/quentin/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) Creating SSH key...
(minikube) Fix file permission...
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/boot2docker.iso
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/config.json
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/id_rsa
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/id_rsa.pub
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/initrd.img
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/vmlinuz64
(minikube) Generate UUID...
(minikube) DBG | Generated UUID: 99C7DD29-AAD5-4633-9E15-4DE3F8BBCC97
(minikube) Convert UUID to MAC address...
(minikube) DBG | Converted MAC address: 4a:86:6b:3c:1a:09
(minikube) Starting minikube...
(minikube) DBG | [xhyve -A -U 99C7DD29-AAD5-4633-9E15-4DE3F8BBCC97 -c 1 -m 1024M -l com1,autopty -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -s 3,ahci-cd,/Users/quentin/.minikube/machines/minikube/boot2docker.iso -s 4:0,ahci-hd,/dev/rdisk4 -f kexec,/Users/quentin/.minikube/machines/minikube/vmlinuz64,/Users/quentin/.minikube/machines/minikube/initrd.img,loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 base host=minikube -F /Users/quentin/.minikube/machines/minikube/minikube.pid -s 5,virtio-9p,host=/Users]
(minikube) Waiting for VM to come online...
(minikube) DBG |
(minikube) DBG | Not there yet 1/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG | operation not supported by device
(minikube) Waiting on a pseudo-terminal to be ready... done
(minikube) Hook up your terminal emulator to /dev/ttys003 in order to connect to your VM
(minikube) DBG |
(minikube) DBG | Not there yet 2/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 3/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 4/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG | rdmsr to register 0x34 on vcpu 0
(minikube) DBG |
(minikube) DBG | Not there yet 5/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 6/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 7/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 8/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 9/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 10/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
...
...
...
(minikube) DBG |
(minikube) DBG | Not there yet 57/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 58/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 59/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 60/60, error: IP not found for MAC 4a:86:6b:3c:1a:09 in DHCP leases
E1018 14:56:42.882671   73832 start.go:87] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds, aborting. Retrying.
I1018 14:56:42.883546   73832 cluster.go:75] Machine exists!
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:61953
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
I1018 14:56:42.898091   73832 cluster.go:82] Machine state:  Running
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
...
...
...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
E1018 14:59:42.986279   73832 start.go:87] Error starting host: Error configuring auth on host: %s: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded. Retrying.
I1018 14:59:42.986522   73832 cluster.go:75] Machine exists!
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:61978
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetState
I1018 14:59:43.006596   73832 cluster.go:82] Machine state:  Running
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) DBG |
Error getting ssh command 'exit 0' : IP not found for MAC  in DHCP leases
...
...
...

As you can see from the shortened logs, this does not work.

What you expected to happen:

I expected cluster creation to work. What's strange is that it does work after installing Docker for Mac using brew cask install docker (and not brew install docker). But it's the same Docker version!

$ docker -v
Docker version 1.12.2, build bb80604
$ minikube delete
Deleting local Kubernetes cluster...
Machine deleted.
$ minikube start --vm-driver=xhyve --show-libmachine-logs --v=10 --alsologtostderr
I1018 15:02:49.748996   74634 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
Found binary path at /usr/local/bin/docker-machine-driver-xhyve
Launching plugin server for driver xhyve
Plugin server listening at address 127.0.0.1:62023
() DBG | operation not supported by device
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) DBG | ===== Docker Machine xhyve Driver Version 0.2.3 (Homebrew) =====
(minikube) DBG |
(minikube) DBG | executing: /usr/local/bin/VBoxManage -v
(minikube) DBG | STDOUT: 5.0.4r102546
(minikube) DBG |
(minikube) DBG | STDERR:
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Downloading /Users/quentin/.minikube/cache/boot2docker.iso from file:///Users/quentin/.minikube/cache/iso/minikube-0.7.iso...
(minikube) Creating VM...
(minikube) Extracting vmlinuz64 and initrd.img from boot2docker.iso...
(minikube) DBG | Mounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil attach /Users/quentin/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/quentin/.minikube/machines/minikube/b2d-image] []  <nil> 0xc42008e008 0xc42008e010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} attach /Users/quentin/.minikube/machines/minikube/boot2docker.iso -mountpoint /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) /dev/disk5                                           /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) DBG | Extracting vmlinuz64 into /Users/quentin/.minikube/machines/minikube
(minikube) DBG | Extracting initrd.img into /Users/quentin/.minikube/machines/minikube
(minikube) DBG | Unmounting boot2docker.iso
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil detach /Users/quentin/.minikube/machines/minikube/b2d-image] []  <nil> 0xc42008e008 0xc42008e010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} detach /Users/quentin/.minikube/machines/minikube/b2d-image
(minikube) "disk5" unmounted.
(minikube) "disk5" ejected.
(minikube) Generating 20000MB disk image...
(minikube) DBG | executing: &{/usr/bin/hdiutil [hdiutil create -megabytes 20000 -type SPARSEBUNDLE /Users/quentin/.minikube/machines/minikube/root-volume] []  <nil> 0xc42008e008 0xc42008e010 [] <nil> <nil> <nil> <nil> <nil> false [] [] [] [] <nil> <nil>} create -megabytes 20000 -type SPARSEBUNDLE /Users/quentin/.minikube/machines/minikube/root-volume
(minikube) created: /Users/quentin/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) Creating SSH key...
(minikube) Fix file permission...
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/boot2docker.iso
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/config.json
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/id_rsa
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/id_rsa.pub
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/initrd.img
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/root-volume.sparsebundle
(minikube) DBG | /Users/quentin/.minikube/machines/minikube/vmlinuz64
(minikube) Generate UUID...
(minikube) DBG | Generated UUID: 33748FBA-E4D9-48FA-91AC-8623EC32479B
(minikube) Convert UUID to MAC address...
(minikube) DBG | Converted MAC address: 12:69:c2:1a:41:8b
(minikube) Starting minikube...
(minikube) DBG | [xhyve -A -U 33748FBA-E4D9-48FA-91AC-8623EC32479B -c 1 -m 1024M -l com1,autopty -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -s 3,ahci-cd,/Users/quentin/.minikube/machines/minikube/boot2docker.iso -s 4:0,ahci-hd,/dev/rdisk5 -f kexec,/Users/quentin/.minikube/machines/minikube/vmlinuz64,/Users/quentin/.minikube/machines/minikube/initrd.img,loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 base host=minikube -F /Users/quentin/.minikube/machines/minikube/minikube.pid -s 5,virtio-9p,host=/Users]
(minikube) Waiting for VM to come online...
(minikube) DBG |
(minikube) DBG | Not there yet 1/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG | operation not supported by device
(minikube) Waiting on a pseudo-terminal to be ready... done
(minikube) Hook up your terminal emulator to /dev/ttys003 in order to connect to your VM
(minikube) DBG |
(minikube) DBG | Not there yet 2/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 3/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 4/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG | rdmsr to register 0x34 on vcpu 0
(minikube) DBG |
(minikube) DBG | Not there yet 5/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 6/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 7/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 8/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 9/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 10/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 11/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 12/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 13/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG |
(minikube) DBG | Not there yet 14/60, error: IP not found for MAC 12:69:c2:1a:41:8b in DHCP leases
(minikube) DBG | 192.168.64.8
(minikube) DBG | IP found in DHCP lease table: 192.168.64.8
(minikube) DBG | Got an ip: 192.168.64.8
(minikube) DBG | Getting to WaitForSSH function...
(minikube) DBG | Using SSH client type: external
(minikube) DBG | {[-o BatchMode=yes -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.8 -o IdentitiesOnly=yes -i /Users/quentin/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh}
(minikube) DBG | About to run SSH command:
(minikube) DBG | exit 0
(minikube) DBG | SSH cmd err, output: <nil>:
(minikube) DBG | Using SSH client type: external
(minikube) DBG | {[-o BatchMode=yes -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.8 -o IdentitiesOnly=yes -i /Users/quentin/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh}
(minikube) DBG | About to run SSH command:
(minikube) DBG | echo -e "#/bin/bash\nsudo mkdir -p /Users\nsudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=quentin -o dfltuid=1000 -o dfltgid=50 -o access=any host /Users" | sudo tee /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh
(minikube) DBG | SSH cmd err, output: <nil>: #/bin/bash
(minikube) DBG | sudo mkdir -p /Users
(minikube) DBG | sudo mount -t 9p -o version=9p2000 -o trans=virtio -o uname=quentin -o dfltuid=1000 -o dfltgid=50 -o access=any host /Users
(minikube) DBG |
(minikube) Calling .GetConfigRaw
(minikube) Calling .DriverName
(minikube) Calling .DriverName
Waiting for machine to be running, this may take a few minutes...
(minikube) Calling .GetState
Detecting operating system of created instance...
Waiting for SSH to be available...
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/quentin/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.8 -o IdentitiesOnly=yes -i /Users/quentin/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: <nil>:
Detecting the provisioner...
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /Users/quentin/.minikube/machines/minikube/id_rsa (-rw-------)
&{[-F /dev/null -o PasswordAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none docker@192.168.64.8 -o IdentitiesOnly=yes -i /Users/quentin/.minikube/machines/minikube/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
cat /etc/os-release
SSH cmd err, output: <nil>: NAME=Boot2Docker
VERSION=1.11.1
ID=boot2docker
ID_LIKE=tcl
VERSION_ID=1.11.1
PRETTY_NAME="Boot2Docker 1.11.1 (TCL 7.1); master : 901340f - Fri Jul  1 22:52:19 UTC 2016"
ANSI_COLOR="1;34"
HOME_URL="http://boot2docker.io"
SUPPORT_URL="https://github.com/boot2docker/boot2docker"
BUG_REPORT_URL="https://github.com/boot2docker/boot2docker/issues"

found compatible host: boot2docker
Provisioning with boot2docker...

...
...
...

(I think it would have worked with Docker Toolbox too.)

How to reproduce it (as minimally and precisely as possible):

Update: While the steps were completely reproducible for me, they no longer are, see https://github.com/kubernetes/minikube/issues/708#issuecomment-254779233

  1. brew cask uninstall docker
  2. brew install docker
  3. minikube delete
  4. minikube start --vm-driver=xhyve --show-libmachine-logs --v=10 --alsologtostderr

Anything else do we need to know:

mboersma commented 8 years ago

I'm having the same problem described above, as I've noted in #646. But I've tried both brew install docker and brew cask install docker and neither one works well with minikube + xhyve on my Mac.

dlorenc commented 8 years ago

@zchee do you think this is the same UUID issue you've mentioned a few other times?

0ff commented 8 years ago

With xhyve + docker for mac, I can ping 8.8.8.8 and google.com successfully, but I can't curl google.com: it timeouts (separate issue but I thought I should mention it).

@pquentin The exact same thing happens for me - for some reason, minikube decides to use the host as the DNS Server in this setup. See the /etc/resolv.conf file in your minikube instance (i.e. minikube ssh); if you change this (or run dnsmasq on your host and bind it to your minikube's ip) it'll work just fine.

pquentin commented 8 years ago

@0ff Changed it to 8.8.8.8 but there's no change. I don't think it was using the host ip on my network though. What's surprising too is that I can ping google.com, and this requires DNS resolution, right?

Hmm, so I just managed to create a minikube with xhyve + docker (not for mac). I think the only thing I did differently was to remove ~/.minikube. This appears to confirm what @mboersma said, Docker for Mac does not seem to actually change things here, so I'll remove that from the title.

zchee commented 8 years ago

@dlorenc maybe yes. I'll detail commnet later.

but short answer:

  1. vmnet write some macOS internal files and convert hardcoded UUID
  2. if kill -9 to xhyve-driver(just example), but not cleanup internal files
  3. After that, vmnet does not convert (or use) same UUID because exists junk(before) UUID data in macOS internal files.
jimmidyson commented 8 years ago

The reason for the hardcoded UUID was to try to get a consistent IP across restarts. Do you think that is still going to be possible?

zchee commented 8 years ago

@jimmidyson Yeah, I know. But vmnet interface is not simply. actually, I had same issue at previous. So now test and use disabled use hardcoded UUID version.(means, self build minikube binary) maybe, it solved remove /var/db/dhcpd_leases and /Library/Preferences/SystemConfiguration/com.apple.vmnet.plist, but very risky for end-users.

Now I consider and research solution...

zchee commented 8 years ago

And just question. How did you decide UUID? use uuidgen?

jimmidyson commented 8 years ago

I didn't... @chirino did iirc.

zchee commented 8 years ago

@jimmidyson @dlorenc It might not relate, but macOS have original(?) UUID rules. (I think and maybe)


When I use satori/go.uuid(or etc. google/uuid is same) for generate UUID, sometimes test is fail. It seems that according to the RFC 4122, but vmnet.framework has recognized it as invalid.

This is why I have to generate a uuid using cgo. https://github.com/zchee/docker-machine-driver-xhyve/blob/master/xhyve/uuid.go

I think control of the UUID is should carefully. But, as https://github.com/zchee/docker-machine-driver-xhyve#does-not-clean-up-the-vmnet-when-remove-a-vm, I don't yet understand vmnet.framework internal.

If we want to the same IP with no bugs, needs cleanup /var/db/dhcpd_leases and /Library/Preferences/SystemConfiguration/com.apple.vmnet.plist(or etc). and should be check valid(duplicate?) UUID process.


Edit: uuidgen command line source Sierra version: http://opensource.apple.com/source/xnu/xnu-3248.60.10/libkern/uuid/uuid.c

jimmidyson commented 8 years ago

@zchee You're far more knowledgeable than me on this! I'm not even a mac user... but I'm really grateful for your investigation as it's affecting a number of users. I'm sure we can find quite a few testers of any fixes you propose.

r2d4 commented 8 years ago

@zchee the static UUID in minikube was created using uuidgen on OS X

chirino commented 8 years ago

Note that in the original issue reported is not using the static UUID. Logging is different when the static UUID is used.

(minikube) Generate UUID...
(minikube) DBG | Generated UUID: 99C7DD29-AAD5-4633-9E15-4DE3F8BBCC97

when the static UUID is used, you will see this in the logs instead.

(minikube) Using Supplied UUID: 57FD2012-FA4A-4FF7-AEFF-26E1A1D76847
zchee commented 8 years ago

@jimmidyson oh, thanks :) I'll do my best.

@chirino Yes, and other issue thread have same bug. Now I research it.

I'll create repository that sample and simple UUID convert script written Go later.

zchee commented 8 years ago

@r2d4 Got it. thanks.

maybe caused by vmnet internal process or conver(search) IP from dhcpd_leases. I'll debug it.

chirino commented 8 years ago

FYI i've run into a similar issue in the past. What I did wrong was trying to build docker-machine-driver-xhyve using plain go get. Builds fine but it does not work. You HAVE to build with make.

zchee commented 8 years ago

@chirino Ah, good information. I'll also check the go get plain build binary behavior.

zchee commented 8 years ago

@pquentin and small reply

With xhyve + docker for mac, I can ping 8.8.8.8 and google.com successfully, but I can't curl google.com: it timeouts (separate issue but I thought I should mention it).

It means use Docker for Mac only? If so, It might solve with

System Preferences -> Network -> Location -> Edit Location... -> +(create clean setting)

It might is associated as well with this problem.

pquentin commented 8 years ago

@zchee Thanks! It does not appear to help though. I hope I did that correctly.

As I was saying, it turns out that there's no longer a difference between Docker for Mac and Docker. Maybe because I now remove ~/.minikube each time. (I also hope that it was really Docker and Docker for Mac above, because docker -v now returns something different for the two versions. I can produce the logs again if that would help.)

The minikube creation itself is no longer a problem for me: I would be happy to close this issue, but it looks like it's now bigger than me. :)

mboersma commented 8 years ago

You HAVE to build [docker-machine-driver-xhyve] with make

I've been doing this as @zchee suggested.

I would be happy to close this issue, but it looks like it's now bigger than me.

Unfortunately, yes, at least for me and a couple coworkers. Using xhyve and minikube v0.12.0 ends up here:

$ minikube start --logtostderr --show-libmachine-logs --vm-driver=xhyve
I1024 14:19:39.583187   22679 notify.go:111] Checking for updates...
Starting local Kubernetes cluster...
I1024 14:19:39.816177   22679 cluster.go:79] Machine exists!
I1024 14:19:39.836294   22679 cluster.go:86] Machine state:  Running
Waiting for SSH to be available...
E1024 14:22:39.968309   22679 start.go:90] Error starting host: Error configuring auth on host: %s: Temporary Error: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded. Retrying.
E1024 14:22:39.968418   22679 start.go:96] Error starting host:  Error configuring auth on host: %s: Temporary Error: Too many retries waiting for SSH to be available.  Last error: Maximum number of retries (60) exceeded

I've also tried with and without the "regenerate UUID" patch to the xhyve driver, but it doesn't seem to matter.

zchee commented 8 years ago

@mboersma Sorry for the delay.

I was some debug and research vmnet.framework internal. I understand some of the internal implementation. but still not understand causes.

However, found a good debugging way. AFAIK, If you already disable SIP macOS feature(because not handled some error if enable), you can get a useful debug log in it.

open -a Console.app

and re-try minikube start .... and visual-grep(check with your eyes) to Console.app log...

If you are confused by many of the logs, add to search dialog.

subsystem: com.apple.network

Please try it and tell me the error log.

In my case, I was tested case of have not sudo privilege with enabling virtio-net. (use original xhyve command, not use docker-machine-driver-xhyve and minikube)

Ultimately, I found that internet sharing was processed in the

xhyve -A -U 87ADA66A-775D-4070-8DEA-B83AB3A47C60 -l com1,stdio -s 0:0,hostbridge -s 31,lpc -s 2:0,virtio-net -f 'kexec,/path/to/vmlinuz,/path/to/initrd,"earlyprintk=serial console=ttyS0 bootmem_debug debug loglevel=7'
            |
/System/Library/Frameworks/vmnet.framework
            | 
/System/Library/PrivateFrameworks/Netrb.framework
bhundven commented 8 years ago

I have the same output as @mboersma.

adamreese commented 7 years ago

Here is my console log https://gist.github.com/adamreese/a82c10b39b438de7328cbc7f475aa61f

zchee commented 7 years ago

@adamreese Thanks for detail log! It seems failed nw_interface_create_with_name in the libsystem_network.dylib. I'll debug it and read https://opensource.apple.com/.

puja108 commented 7 years ago

Like I already mentioned in https://github.com/kubernetes/minikube/issues/646 this is now solved in my system with some latest updates. Not sure which one actually fixed it, but my current versions are:


minikube version: v0.16.0
docker-machine version 0.9.0, build 15fd4c7
docker-machine-driver-xhyve: stable 0.3.1 (bottled), HEAD
xhyve: stable 0.2.0 (bottled), HEAD```
andyp1per commented 7 years ago

It's not working for me. In Console.app I see:

docker-machine-driver-xhyve[982] invalid client error: aborting XPC connection to daemon interface 0x7000600, netrb client create failed

in the minikube log:

(minikube) Convert UUID to MAC address... E0515 14:12:41.673413 1746 start.go:119] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Could not convert the UUID to MAC address: exit status 1.

maduramarathe9 commented 7 years ago

Hi,

I followed these steps:

  1. brew install kubectl
  2. brew cask install minikube
  3. brew install docker-engine-driver=xhyve
  4. minikube start --vm-driver=xhyve

minikube failed to start

" $ minikube start --vm-driver=xhyve Starting local Kubernetes v1.7.0 cluster... Starting VM... Downloading Minikube ISO 97.80 MB / 97.80 MB [==============================================] 100.00% 0s E0822 23:28:00.726776 57157 start.go:129] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Could not convert the UUID to MAC address: exit status 1.

Retrying. E0822 23:28:00.727478 57157 start.go:135] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Could not convert the UUID to MAC address: exit status 1

An error has occurred. Would you like to opt in to sending anonymized crash information to minikube to help prevent future errors? To opt out of these messages, run the command: minikube config set WantReportErrorPrompt false

Please enter your response [Y/n]: y

"

Please let me know where I am going wrong.

maduramarathe9 commented 7 years ago

rm -rf ~/.minikube sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

minikube start --vm-driver=xhyve

This helped resolve my issue:

$ minikube start --vm-driver=xhyve Starting local Kubernetes v1.7.0 cluster... Starting VM... Downloading Minikube ISO 97.80 MB / 97.80 MB [==============================================] 100.00% 0s Getting VM IP address... Moving files into cluster... Setting up certs... Starting cluster components... Connecting to cluster... Setting up kubeconfig... Kubectl is now configured to use the cluster.

pkbelidha commented 7 years ago

$kubectl run hello-minikube --image=grc.io/google_container/echoserver:1.4 --port=8080

Error: failed to discover supported resources: Get https://192.168.99.100:8443/api: dial tcp 192.168.99.100:8443: i/o timeout

Can you please help me how to resolve this ?

ankurkumarz commented 7 years ago

Yes it worked for me as well.

Please follow the same order.

sudo rm -rf ~/.minikube sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve minikube start --vm-driver=xhyve

joeljohnston commented 6 years ago

I'm having this same problem with both minikube and docker-machine using the docker-machine-driver-xhyve 0.33 release. The problem began happening after i upgraded from minikube 0.25.0 to minikube 0.25.2. The solution above does not resolve the issue. I've emptied the dhcpd_leases file and ensured permissions are set appropriately to no avail. Please help. Thanks.