search

docker / machine

Machine management for a container-centric world
https://docs.docker.com/machine/
Apache License 2.0
6.63k stars 1.97k forks source link

virtualbox: Hangs on Waiting for VM to start #986

Closed dave-tucker closed 9 years ago

dave-tucker commented 9 years ago

Tried starting a VM via docker-machine start and it hung on:

INFO[0000] Waiting for VM to start...

Check the VirtualBox GUI and the VM state was aborted. The underlying cause here was that my VM image was corrupted, but as a user I'd expect machine to be aware of this and provide an appropriate error message.

ehazlett commented 9 years ago

Hmm I haven't seen that before -- thanks for reporting. How were you able to tell it was corrupted?

dave-tucker commented 9 years ago

I was guessing at "corrupted". On manual start of the VM from the virtualbox GUI it got mid-way through the boot process and then the state changed to "aborted". A little digging suggests it could be related to a bug in virtualbox....

Still, it would be great for machine to notify the user about these type of errors when they occur.

ms-clovis commented 9 years ago

This happens in with boot2docker image also. If you stop the vm for the image and choose stop (instead of option sending signal to stop, it will show aborted and probably not restart from virtual machine or boot2docker cli. Recommended fix (and it worked for me) is simply rebooting windows or whatever os you are running boot2docker on. Hope this helps.

sshaw commented 9 years ago

Same problem after installing from https://github.com/docker/machine/releases/download/v0.2.0/docker-machine_darwin-amd64:

~ >docker-machine create --driver virtualbox dev
INFO[0000] Creating CA: /Users/sshaw/.docker/machine/certs/ca.pem
INFO[0000] Creating client certificate: /Users/sshaw/.docker/machine/certs/cert.pem
INFO[0001] Creating SSH key...
INFO[0001] Image cache does not exist, creating it at /Users/sshaw/.docker/machine/cache...
INFO[0001] No default boot2docker iso found locally, downloading the latest release...
INFO[0001] Downloading latest boot2docker release to /Users/sshaw/.docker/machine/cache/boot2docker.iso...
INFO[0024] Creating VirtualBox VM...
INFO[0034] Starting VirtualBox VM...
INFO[0035] Waiting for VM to start...

VBoxHeadless is running.

In my case opening VirtualBox shows the dev image as Running and the docker ASCII art + command prompt in the Preview section.

sshaw commented 9 years ago

Also note that the aforementioned bug was supposed to be fixed in 4.3.24 but I was running 4.3.25. Tried with 4.3.26 too, same problem.

DavideDelVecchio commented 9 years ago

Same thing here I'm on

docker-machine create --driver virtualbox mymachine

it hangs

ehazlett commented 9 years ago

@sshaw @DavideDelVecchio can you post the output of both VBoxManager list hostonlyifs and netstat -nr?

sshaw commented 9 years ago 
~ >VBoxManage list hostonlyifs
Name:            vboxnet0
GUID:            786f6276-656e-4074-8000-0a0027000000
DHCP:            Disabled
IPAddress:       192.168.99.1
NetworkMask:     255.255.255.0
IPV6Address:
IPV6NetworkMaskPrefixLength: 0
HardwareAddress: 0a:00:27:00:00:00
MediumType:      Ethernet
Status:          Up
VBoxNetworkName: HostInterfaceNetworking-vboxnet0
~ >netstat -nr
Routing tables
Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.1.1        UGSc           25        1     en0
127                127.0.0.1          UCS             0      126     lo0
127.0.0.1          127.0.0.1          UH              2  1772668     lo0
127.94.0.1         127.94.0.1         UH              0        0     lo0
127.94.0.2         127.94.0.2         UH              0        0     lo0
192.168.1          link#4             UCS             1        0     en0
...
192.168.1.130      127.0.0.1          UHS             0       88     lo0
192.168.99         link#9             UC              1        0 vboxnet

Note that deleting the route and replacing it with:

route add -net  192.168.99 -interface vboxnet0

still results in a route to the interface vboxnet.

Maybe this is the problem?

pugnascotia commented 9 years ago

I saw the same behaviour last night - I installed the latest VirtualBox from the official website, installed docker and docker-machine via brew, and it hung on "Waiting for VM to start". I didn't get a chance to try rebooting the machine.

DavideDelVecchio commented 9 years ago

I finally got it work this way I have previously installed Kitematic that hanged upon startup installed then docker-machine via brew and had "Waiting for VM to start" then removed all the VMs and installed docker-machine this way curl -L https://github.com/docker/machine/releases/download/v0.2.0/docker-machine_darwin-amd64 > /usr/local/bin/docker-machine chmod +x /usr/local/bin/docker-machine docker-machine create --driver virtualbox mymachine I have all my route on interface on vboxnet as well

ehazlett commented 9 years ago

@sshaw thanks for the feedback. if the route (as shown in the table) is using vboxnet and not the vboxnet0 interface, this looks like the issue. i was able to reproduce this behavior by manually removing my routing entries. i'm not sure why this would be messed with -- are you using any VPN software by chance? @pugnascotia are you using vpn?

pugnascotia commented 9 years ago

Nope, no VPN for me. I'll try to dig into this more.

sshaw commented 9 years ago

@ehazlett yes, seems so but when I delete the route and do:

route add -net  192.168.99 -interface vboxnet0

The route gets added as vboxnet. Does this happen for you -or anyone else?

ehazlett commented 9 years ago

@sshaw the way i was to fix was to delete the route and then ifconfig vboxnet0 down and ifconfig vboxnet0 up. the route was back up and i was able to connect.

ehazlett commented 9 years ago

@sshaw ok this is a difference between OS X and Linux -- on Linux it shows as the exact name but on OS X i see it as vboxnet.

sshaw commented 9 years ago

Ah yes, I thought you were testing on OS X too...

mattes commented 9 years ago

+1

robtec commented 9 years ago

@sshaw can confirm deleting vboxnet then adding the vboxnet0 results in vboxnet present, again

robtec commented 9 years ago

So before I added a VM via boot2docker

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.0.1        UGSc           72        1     en1
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH            256    17176     lo0
169.254            link#5             UCS             0        0     en1
192.168.0          link#5             UCS             1        0     en1
192.168.0.1/32     link#5             UCS             1        0     en1
192.168.0.1        c4:27:95:b5:13:9   UHLWIir        74       24     en1   1141
192.168.0.15/32    link#5             UCS             0        0     en1
192.168.0.255      ff:ff:ff:ff:ff:ff  UHLWbI          0        1     en1
192.168.99

and after...

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
default            192.168.0.1        UGSc           34        7     en1
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH              3    21645     lo0
169.254            link#5             UCS             0        0     en1
192.168.0          link#5             UCS             0        0     en1
192.168.0.1/32     link#5             UCS             1        0     en1
192.168.0.1        c4:27:95:b5:13:9   UHLWIir        36       24     en1    383
192.168.0.15/32    link#5             UCS             0        0     en1
192.168.59         link#11            UC              1        0 vboxnet
192.168.59.103     8:0:27:bc:84:0     UHLWIi          1       19 vboxnet   1196
192.168.99         link#10            UCSc            1        0 vboxnet

The new 192.168.59.103 entry looks different to the 192.168.99

Hope this helps :+1:

robcowie commented 9 years ago

Does anyone have a definitive explanation or solution to this? It's become an issue for me and I know so little about virtualbox and docker that I an unable proceed.

ehazlett commented 9 years ago

@robcowie unfortunately we are still investigating. it appears to be how virtualbox is handling multiple host only interfaces. i've been able to debug where virtualbox will not give out an IP address via its built in DHCP when there are multiple host only interfaces. i would try removing all of the the host only interfaces and then re-run machine.

ljrittle commented 9 years ago

FWIW, I tried this: " i would try removing all of the the host only interfaces and then re-run machine." with machine-0.2 and vbox 4.3.26 on an OS X machine without luck.

@robcowie https://github.com/robcowie unfortunately we are still

investigating. it appears to be how virtualbox is handling multiple host only interfaces. i've been able to debug where virtualbox will not give out an IP address via its built in DHCP when there are multiple host only interfaces. i would try removing all of the the host only interfaces and then re-run machine.

craigbrad commented 9 years ago

Any update on this? I'm having the same problem

nagendersoma commented 9 years ago

I just tried docker-machine for the first time (OSX, virtualbox) and hit this issue. Running following cmds resolved the issue. ifconfig vboxnet0 down ifconfig vboxnet0 up

One observation : when I use vagrant to configure new VMs, it uses adapter-2 host-only interface with adpater-type Intel PRO/1000MT. The vm creation goes smooth. With docker-machines, I see host-only adapter-type is set to paravirtualized-virtio.
Is this difference potentially causing any issues ?

sflxn commented 9 years ago

Just wanted to leave my experience with this issue. I was having this problem on my Macbook Air but not on my Macbook Pro. I assume it is some software configuration. Anyway, I pulled the latest code on my Macbook Air and built it. I had no problems with the latest code. I have not compared the difference between the code. Btw, when I pulled the latest code, I had to delete the Azure and DigitialOcean drivers before I could build it.

ehazlett commented 9 years ago

@nagendersoma thanks for the feedback!

we've narrowed it down to the hostonlyif and the VirtualBox dhcp service.

@craigbrad @robcowie @pugnascotia could you try changing your interface type to Intel PRO/1000MT and see if that works any better?

adamrothman commented 9 years ago

@ehazlett I've tried setting Adapter 2 to Intel PRO/1000 MT (both Desktop and Server) with no luck. docker-machine start docker-vm hangs on Waiting for VM to start... as before.

For what it's worth, this is happening on both a Mac mini and MacBook Pro. Machines that I created about 2 weeks ago seem to be fine. docker-machine hasn't updated since I was last able to use it successfully. VirtualBox has – from 4.3.26 to 4.3.28. Maybe that has something to do with it?

adamrothman commented 9 years ago

Confirmed that completely uninstalling VirtualBox 4.3.28 (using the .tool in the DMG) and installing VirtualBox 4.3.26 fixes the issue. Weird.

adamrothman commented 9 years ago

Just kidding. Downgrading to VirtualBox 4.3.26 only fixed the issue on my MacBook Pro. Still not working on my Mac mini.

I tried this:

$ route delete -net 192.168.99
$ ifconfig vboxnet0 down
$ ifconfig vboxnet0 up

No luck.

adamrothman commented 9 years ago

Building the latest docker-machine from source, restarting the Mac mini, and re-creating the image worked.

craigbrad commented 9 years ago

@ehazlett I had no luck with that, however I did manage to get it working eventually by removing all of my host only interfaces and routes then creating a new machine. It seemed to take a while to connect though. I'm on VB 4.3.28.

ehazlett commented 9 years ago

@adamrothman this would make sense -- we've been able to narrow the issue to the vbox dhcp server which would make sense.

efrecon commented 9 years ago

I also have started to run into this issue. It's intermittent, running in debug mode seems to help sometimes (but nothing 100%). Is there any way to circumvent the issue, restarting the VirtualBox DHCP server or similar? I am trying to automate this from machinery and would prefer squeezing that bug away before the next release.

efrecon commented 9 years ago

For what it is worth, I thought that I would mention some of the steps that I have taken while trying to sort this out. It turned out that I had both an old virtual box DHCP server running and an old vboxnetX interface lying around (probably in part because I have run vagrant on the same machine). Getting rid of those seems to help a little bit.

To get the list of running DHCP servers, you can run:

VBoxManage list dhcpservers

And you can remove them through issuing (adapt to the proper vboxnetX interfaces) and repeat as many times as necessary.

VBoxManage dhcpserver remove --netname HostInterfaceNetworking-vboxnet0

In the process, I also removed the old interfaces using (adapt again):

VBoxManage hostonlyif remove vboxnet0
codecakes commented 9 years ago

i removed the dhcpserver as directed above and then added the following:

VBoxManage list dhcpservers

NetworkName:    intnet0
IP:             192.168.66.1
NetworkMask:    255.255.255.0
lowerIPAddress: 192.168.66.2
upperIPAddress: 192.168.66.255
Enabled:        Yes

NetworkName:    HostInterfaceNetworking-vboxnet0
IP:             192.168.99.1
NetworkMask:    255.255.255.0
lowerIPAddress: 192.168.99.2
upperIPAddress: 192.168.99.255
Enabled:        Yes

Haven't restarted the machine though. and its stuck at 

INFO[0000] Creating SSH key...
INFO[0000] Creating VirtualBox VM...
INFO[0015] Starting VirtualBox VM...
INFO[0016] Waiting for VM to start...

Does rebooting help anyone?

efrecon commented 9 years ago

Sometimes... It's all.. well... intermittent.

ehazlett commented 9 years ago

@efrecon could you try this build and see if it's any better? https://public.evanhazlett.com/docker-machine/vbox-intel-nic/

philm commented 9 years ago

@ehazlett - your build worked for me (with Virtualbox 4.3.28 and OS X 10.10.4).

dueyfinster commented 9 years ago

@ehazlett your build worked for me - VirtualBox 4.3.28 / Mac OS X 10.10.3

efrecon commented 9 years ago

@ehazlett, I was able to create a dozen of virtual machines without a glitch with your build (machinery makes this kind of testing easy!). I am on Ubuntu and running VirtualBox 4.3.28 also. I have another Ubuntu-host that has had similar problems in the past, I'll try on that one tomorrow.

Note however that the ssh port to get into the machine is 22 with your build, but is reported as something else by docker-machine inspect. I guess that you know, but I thought that I would mention it just in case.

What is the estimated time frame for the release of 0.3?

adamrothman commented 9 years ago

@ehazlett I just did a fresh install of VirtualBox 4.3.28 and downloaded your build. Got this:

$ docker-machine create --driver virtualbox docker-vm
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
Error creating machine: No IP address found 4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:a4:ca:bc brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a00:27ff:fea4:cabc/64 scope link 
       valid_lft forever preferred_lft forever

You will want to check the provider to make sure the machine and associated resources were properly removed.

Restarting and un/reinstalling VirtualBox did not help.

adamrothman commented 9 years ago

My routing tables got all messed up as a result of these shenanigans so I deleted the bad entries, added the right ones, and everything is working with VirtualBox 4.3.26 so I'm going to leave it alone.

eyakcn commented 9 years ago

I think it's because there exist boot2docker-vm, after I remove all the virtual machines, then everything works

efrecon commented 9 years ago

@ehazlett, your build is also working on the other host that I had problems on. It is running a lower build number of VirtualBox 4.3.28, still on Ubuntu LTS.

codecakes commented 9 years ago

@ehazlett 's build works for me on Ubuntu v 14.04 LTS 64 bit

Chukwunonso commented 9 years ago

@ehazlett just confirm that your build works for me.

CyrilPeponnet commented 9 years ago

I also confirm that the binary from https://public.evanhazlett.com/docker-machine/vbox-intel-nic/docker-machine_darwin-amd64 fixes the issue with vBox 4.3.26 under OSX 10.10.3 (even if I managed to create the first vm with the release binary).

ehazlett commented 9 years ago

Thanks for the feedback all! I'll get that fixed merged!

ehazlett commented 9 years ago

FYI issued #1218 to get that fix merged.

pugnascotia commented 9 years ago

Cool - I've been moving house so I didn't get a chance to try the fix. Great work!

On 22 May 2015 at 15:36, Evan Hazlett notifications@github.com wrote:

FYI issued #1218 https://github.com/docker/machine/pull/1218 to get that fix merged.

— Reply to this email directly or view it on GitHub https://github.com/docker/machine/issues/986#issuecomment-104675868.

Rory Hunter