random "Forcing shutdown of VM" through provisioning process

[LarsFronius]

If I run "vagrant up" and have a puppet provisioner configured, the provisioning process sometimes gets abborted by the following message. I think this happens more often if I run puppet in the foreground - being done with puppet.options = ["--test"] in the Vagrantfile.

[avm01] Forcing shutdown of VM...
[avm01] Destroying VM and associated drives...
[avm01] Destroying unused networking interface...
/usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/ruby_compat.rb:22:in `select': closed stream (IOError)
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/ruby_compat.rb:22:in `io_select'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/transport/packet_stream.rb:73:in `available_for_read?'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/transport/packet_stream.rb:85:in `next_packet'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/transport/session.rb:169:in `block in poll_message'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/transport/session.rb:164:in `loop'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/transport/session.rb:164:in `poll_message'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/connection/session.rb:451:in `dispatch_incoming_packets'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/connection/session.rb:213:in `preprocess'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/connection/session.rb:197:in `process'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/connection/session.rb:161:in `block in loop'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/connection/session.rb:161:in `loop'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/connection/session.rb:161:in `loop'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/net-ssh-2.1.4/lib/net/ssh/connection/channel.rb:269:in `wait'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/ssh/session.rb:55:in `sudo!'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/provisioners/puppet_server.rb:47:in `block in run_puppetd_client'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/ssh.rb:119:in `execute'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/provisioners/puppet_server.rb:46:in `run_puppetd_client'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/provisioners/puppet_server.rb:24:in `provision!'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/provision.rb:22:in `block in call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/provision.rb:20:in `each'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/provision.rb:20:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/warden.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/forward_ports.rb:95:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/warden.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/clear_forwarded_ports.rb:21:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/warden.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/clean_machine_folder.rb:17:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/warden.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/check_guest_additions.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/warden.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/match_mac_address.rb:21:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/warden.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/import.rb:26:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/warden.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/vm/check_box.rb:23:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/warden.rb:30:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action/builder.rb:120:in `call'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action.rb:134:in `block (2 levels) in run'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/util/busy.rb:19:in `busy'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action.rb:134:in `block in run'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/environment.rb:364:in `block in lock'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/environment.rb:354:in `open'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/environment.rb:354:in `lock'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/action.rb:133:in `run'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/vm.rb:140:in `up'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/command/up.rb:13:in `block in execute'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/command/up.rb:8:in `each'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/command/up.rb:8:in `execute'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/task.rb:22:in `run'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/invocation.rb:118:in `invoke_task'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/invocation.rb:124:in `block in invoke_all'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/invocation.rb:124:in `each'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/invocation.rb:124:in `map'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/invocation.rb:124:in `invoke_all'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/group.rb:226:in `dispatch'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/invocation.rb:109:in `invoke'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/lib/vagrant/cli.rb:45:in `block in register'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/task.rb:22:in `run'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/invocation.rb:118:in `invoke_task'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor.rb:263:in `dispatch'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/thor-0.14.6/lib/thor/base.rb:389:in `start'
    from /usr/local/Cellar/ruby/1.9.2-p290/lib/ruby/gems/1.9.1/gems/vagrant-0.8.7/bin/vagrant:21:in `<top (required)>'
    from /usr/local/Cellar/ruby/1.9.2-p290/bin/vagrant:19:in `load'
    from /usr/local/Cellar/ruby/1.9.2-p290/bin/vagrant:19:in `<main>'

I think this doesn't fail when I only run 'vagrant provision' after I created a VM without a provisioner configured in the Vagrantfile when created with 'vagrant up'.

I run ruby 1.9.2 with vagrant 0.8.7 on a Mac OS 10.7.1 in iTerm 2 - if this is relevant for the "closed stream (IOError)" in the first ruby error line. But I have also heard from a colleague who runs Ubuntu, that he has the same issue. He uses puppetfiles from his local machine and I use a remote puppet server. From that point of view I can say that I hardly believe this is a general problem, rather than a Mac OS or iTerm specific or a problem with that I use a remote puppet server.

This is my Vagrantfile btw. Vagrant::Config.run do |config| config.vm.define :avm01 do |config| config.vm.box = "squeeze-pxe-vbox4.1.4-v3" config.vm.network "33.33.33.10" config.vm.customize do |vm| vm.memory_size = 1024 end end config.vm.provision :puppet_server do |puppet| puppet.puppet_server = "puppet.fq.dn" puppet.puppet_node = "avm01.vagrant.internal" puppet.options = ["--test"] end end

[lucaspottersky]

The "`select': closed stream (IOError)" error happens to me whenever I run "vagrant provision" on a host/guest Ubuntu.

I have a very simple recipe that hangs when it tries to "rbenv_gem "rails". I'm unable to provision my vagrant machine. The workaround mentioned (modifyvm nictype1) by @dcarley didn't work for me.

To mitigate this problem, for now I'm simply logging onto the VM and executing Chef-Solo by hand. Totally defeats the purpose of Vagrant, doesn't it? :( Hope one day this gets fixed somehow...

[anentropic]

I agree with @leth 's comment above "I don't think an error during the provisioning step of up should necessarily cause the destruction of the VM"

[leth]

A better workaround: vagrant up --no-provision && vagrant provision.

[anentropic]

@leth when I try that I get the following error:

"Shared folders that Chef requires are missing on the virtual machine. This is usually due to configuration changing after already booting the machine. The fix is to run a vagrant reload so that the proper shared folders will prepared and mounted on the VM."

this comes at the provision stage

[dcarley]

@anentropic What version of Vagrant? There was a change in #803 that made this possible.

[anentropic]

@dcarley I'm on:

$ vagrant --version
Vagrant version 1.0.1

[anentropic]

ok, my version is too old to have that fix...

[anentropic]

cool, works under v1.0.3 :)

although it doesn't alleviate the problem... installing a big pip requirements can still crash the provision step as before, and it often crashes the same way on repeated tries of vagrant provision until you get a run that succeeds.

[jashmenn]

@leth 's solution helped but, like @anentropic, I still have issues when I have a vagrant provision that runs more than a couple of minutes.

[pfalcon]

I had the same issue, while running pretty complex multi-step provision setup. The same as noted by few folks above, suggestion in https://github.com/mitchellh/vagrant/issues/516#issuecomment-3998630 worked for me. While investigating the issue, I also found http://www.virtualbox.org/manual/ch09.html#changenat . Nothing described seems like immediate solution for the problem (I would be looking for something like "NAT port mapping table capacity"), but nonetheless, tweaking network buffer sized and/or DNS modes maybe a good idea for desperate. I didn't have chance to try them, as again, changing NIC type to Am79C973 helped me.

[nand2]

Bitten as well by this (VBox 4.1.24), with long python pip installs; the trick at https://github.com/mitchellh/vagrant/issues/516#issuecomment-3998630 seems to fix things .... so far.

[rb2k]

For me this happens while compiling php. This is after the VM already has been provisioned. The network interface swap didn't fix it. (VBox 4.2.10)

[JamyGolden]

Same problem.

Terminal output: http://pastie.org/7302608 VagrantFile: http://pastie.org/7302638

Operating System: OS X 10.8.3 Vagrant: 1.1.5 VirtualBox: 4.2.10 r84104

Edit: It seems this problem happens on vagrant reload specifically

[mitchellh]

In Vagrant 1.2.0 I send a keep-alive packet every 5 seconds so this should work around this issue. YAY!

[koellcode]

Same problem here on my Macbook Air (WLAN), even on Vagrant 1.2.1 . I'm using OS X 10.8.3 and VirtualBox 4.2.12. After some digging, i've found out that it works over a shared Bluetooth Network Connection via my iPhone, and on my Mac Mini - which is connected directly to Ethernet. I've tried some MTU Size tweaking, but no success.

[mig5]

Like @koellcode this is still a problem for me:

VirtualBox 4.2.12 Vagrant 1.2.1 Debian 6 box

It occurs just trying to run 'apt-get update' once inside the VM (I first encountered the problem when provisioning via Puppet, but found even without the Puppet provisioning, an apt-get update is all it takes to force this shutdown).

I also tried the config.vm.provision NIC modification from @dcarley but it didn't help.

Also makes no difference over wifi or wired ethernet.

[mig5]

FWIW, a new router (or maybe only a reboot of the router) fixed it, which leads me to think it was some sort of network or MTU issue

[koellcode]

like @mig5 i've switched my router too. now it works...

[ctheune]

Ugh. Being bitten as well. Current vagrant and Virtualbox, OS X 10.8.3.

I even saw the box die randomly when logged in with SSH interactively. :(

[henning]

Had the problem yesterday and had to fiddle with Vagrant and Virtualbox for half of the night. Ubuntu Precise 12.04 Vagrant was at 1.22 the whole time. Been using the Virtualbox in Ubuntu (4.1.12), but also Oracle Virtualbox didn't work. It finally worked when switching off VT Extensions in the VM Config. At least once, I currently do not dare to run the process one more time as I fear to be unable to work again if it turns out to happen again... Not sure if that is really a Vagrant or a Virtualbox issue, also.

[aknrdureegaesr]

I had this problem also: Host Debian Wheezy, guest homegrown CentOs 6.4 Vagrant box, both 64 Bit, Vagrant 1.2.3, Virtualbox 4.2.16.

My Vagrant run would execute, via a shell provisioner, the command yum list installed pretty much first thing. This yum list installed hammers the network some trying to find fastesmirror. This yum list installed would also crash about half the time.

Here, "crash" means "Vagrant destroys the virtual machine". Typically last words from such a run:

[default] Destroying VM and associated drives...
/opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/ruby_compat.rb:30:in `select': closed stream (IOError)
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/ruby_compat.rb:30:in `io_select'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/transport/packet_stream.rb:73:in `available_for_read?'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/transport/packet_stream.rb:85:in `next_packet'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/transport/session.rb:172:in `block in poll_message'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/transport/session.rb:167:in `loop'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/transport/session.rb:167:in `poll_message'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/connection/session.rb:454:in `dispatch_incoming_packets'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/connection/session.rb:216:in `preprocess'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/connection/session.rb:200:in `process'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/connection/session.rb:164:in `block in loop'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/connection/session.rb:164:in `loop'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/connection/session.rb:164:in `loop'
    from /opt/vagrant/embedded/gems/gems/net-ssh-2.6.8/lib/net/ssh/connection/channel.rb:269:in `wait'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/communicators/ssh/communicator.rb:318:in `shell_execute'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/communicators/ssh/communicator.rb:61:in `block in execute'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/communicators/ssh/communicator.rb:139:in `connect'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/communicators/ssh/communicator.rb:60:in `execute'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/communicators/ssh/communicator.rb:80:in `sudo'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/provisioners/shell/provisioner.rb:31:in `block (2 levels) in provision'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/provisioners/shell/provisioner.rb:14:in `tap'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/provisioners/shell/provisioner.rb:14:in `block in provision'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/provisioners/shell/provisioner.rb:78:in `with_script_file'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/provisioners/shell/provisioner.rb:12:in `provision'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/provision.rb:65:in `run_provisioner'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/provision.rb:53:in `block in call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/provision.rb:49:in `each'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/provision.rb:49:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/providers/virtualbox/action/clear_forwarded_ports.rb:13:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/providers/virtualbox/action/set_name.rb:35:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/providers/virtualbox/action/clean_machine_folder.rb:17:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/providers/virtualbox/action/check_accessible.rb:18:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/runner.rb:61:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/runner.rb:61:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/call.rb:51:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/runner.rb:61:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/runner.rb:61:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/call.rb:51:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/runner.rb:61:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/runner.rb:61:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/call.rb:51:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/providers/virtualbox/action/check_virtualbox.rb:17:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/call.rb:57:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/plugins/providers/virtualbox/action/check_virtualbox.rb:17:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/warden.rb:34:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/builder.rb:116:in `call'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/runner.rb:61:in `block in run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/util/busy.rb:19:in `busy'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/action/runner.rb:61:in `run'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/machine.rb:147:in `action'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.2.3/lib/vagrant/batch_action.rb:63:in `block (2 levels) in run'

I would appreciate if I could make Vagrant to not destroy the machine, so I can sift through the remaining rubble.

This is a Heisenbug: When I set VAGRANT_LOG=debug, the problem disappears.

I have also been able to once or twice reproduce the crash by removing that shell provisioner from my Vagrantfile and instead using my hosts's native ssh command to connect to the guest, sudo -s there and then execute yum list installed manually. In that case, "crash" means the machine is stopped, according to the Virtualbox GUI.

For me, the problem frequency seemed to decrease when I enlarged VirtualBox's socket buffers from their standard value of 64 kB and also the memory that guest gets:

config.vm.provider "virtualbox" do |vbox|
  vbox.customize ["modifyvm", :id, "--memory", "2048", "--natsettings1", "0,400,400,0,0" ]
end

But, through several attempts at fiddling these values, I could not make the problem go away completely. Also, the provisioning process would frequently fail at a later stage.

How do I re-open this issue?

[koellcode]

It's not fixed yet, so reopening the ticket would be great..

[diranged]

Please re-open this bug. I've seen you're comments in https://github.com/mitchellh/vagrant/issues/2010, but honestly this is still an ongoing issue. We see this about 30% of the time during our Puppet provisioning process. For what its worth, I think in our case this happens when Puppet does a quick reload of the networking stack after making some changes. I'm not sure if that helps diagnose the issue.

Since this is clearly an outstanding issue, I suggest leaving the bug open until it can be resolved. It sends an odd message closing the but without any resolution.

[troygoode]

I think I have a reproducible scenario. This Vagrantfile is meant to stand up an instance of Docker and then load this Dockerfile, but it consistently (100% of the time on my machine after multiple runs) breaks with the error found in this ticket. Tested on Vagrant 1.2.7. Halp?

[mitchellh]

@TroyGoode I was able to reproduce using your setup. Looking into it. THANK YOU!

[troygoode]

:+1: glad my noobishness could be of some help!

[mitchellh]

This is fixed in master.

[troygoode]

great news. thanks @mitchellh. any idea when master'll be cut into a new release?

[mitchellh]

Not at the moment, but "soon"

[bascht]

Just to be sure - did the fix for #516 make it into Vagrant 1.3.2/1.3.3? I couldn't find anything in the Changelog.

[aknrdureegaesr]

I experience this problem with Vagrant 1.3.3, VirtualBox 4.2.18 r88780 on Debian Wheezy.

[malvim]

Same problem here, same versions as @aknrdureegaesr, on OSX 10.8.4.

[make]

Maybe this has something to do with https://github.com/net-ssh/net-ssh/issues/102.

[aknrdureegaesr]

@make: Could be. Two but-s:

But 1 The Vagrant + Virtualbox Setup that showed the problem didn't do anything multithreading, at least none I'm aware of.

But 2 My previous observation:

I have also been able to once or twice reproduce the crash by removing that shell provisioner from my Vagrantfile and instead using my hosts's native ssh command to connect to the guest, sudo -s there and then execute yum list installed manually.

I may be wrong, but my finger tentatively points to the Virtualbox side of things. Virtualbox may have a race condition in its network code.

Personally, I have since moved over to KVM, controlled by home-grown-rake. (If Vagrant works for you, that's not a path you want to follow. But, on the other hand, it is navigable.)

[arnaudlawson]

We have been experiencing a very similar issue. We upgraded from vagrant 1.1.0 to 1.2.2 (running VitualBox 4.2.16r86992) and ever since the upgrade whenever we try to start up a VM that has a second (eth1) host-only NIC adapter vagrant throws an error while provisioning this second NIC: "[default] Configuring and enabling network interfaces... /opt/vagrant/embedded/lib/ruby/1.9.1/tempfile.rb:346:in `rmdir': No such file or directory - /tmp/vagrant20131017-30789-11m1r7h.lock (Errno::ENOENT)" This never happened before the upgrade (so we even degraded back to vagrant 1.1.0 but then experienced the same issue so now we are back to using vagrant 1.2.2 with the same issue). Any help on how to resolve this issue [[Vagrant fails during the provisioning of the second host-only Virtual NIC]] would help. Thanks.

Here is the stack trace.
root@centos5.os.com:/opt/sociocast/operations/vagrant/vms/Centos64BoxyHostonly_test] vagrant reload [default] Attempting graceful shutdown of VM... [default] Setting the name of the VM... [default] Clearing any previously set forwarded ports... [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Preparing network interfaces based on configuration... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Configuring and enabling network interfaces... /opt/vagrant/embedded/lib/ruby/1.9.1/tempfile.rb:346:in rmdir': No such file or directory - /tmp/vagrant20131017-30789-11m1r7h.lock (Errno::ENOENT) from /opt/vagrant/embedded/lib/ruby/1.9.1/tempfile.rb:346:inrmdir' from /opt/vagrant/embedded/lib/ruby/1.9.1/tempfile.rb:338:in ensure in locking' from /opt/vagrant/embedded/lib/ruby/1.9.1/tempfile.rb:338:inlocking' from /opt/vagrant/embedded/lib/ruby/1.9.1/tempfile.rb:144:in block in initialize' from /opt/vagrant/embedded/lib/ruby/1.9.1/tmpdir.rb:133:increate' from /opt/vagrant/embedded/lib/ruby/1.9.1/tempfile.rb:134:in initialize' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/guests/redhat/cap/configure_networks.rb:36:innew' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/guests/redhat/cap/configure_networks.rb:36:in block in configure_networks' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/guests/redhat/cap/configure_networks.rb:21:ineach' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/guests/redhat/cap/configure_networks.rb:21:in configure_networks' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/guest.rb:130:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/guest.rb:130:in capability' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/network.rb:115:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/clear_network_interfaces.rb:26:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/share_folders.rb:25:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/clear_shared_folders.rb:12:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/prepare_nfs_settings.rb:11:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/nfs.rb:28:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/prune_nfs_exports.rb:15:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:118:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/prepare_forwarded_port_collision_params.rb:30:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/env_set.rb:19:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/provision.rb:45:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/clear_forwarded_ports.rb:13:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/set_name.rb:35:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/clean_machine_folder.rb:17:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/check_accessible.rb:18:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:inblock in run' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/util/busy.rb:19:in busy' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:inrun' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/call.rb:51:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:in block in run' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/util/busy.rb:19:inbusy' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:in run' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/call.rb:51:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:inblock in run' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/util/busy.rb:19:in busy' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:inrun' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/call.rb:51:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/config_validate.rb:25:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/check_virtualbox.rb:17:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/call.rb:57:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/check_virtualbox.rb:17:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/config_validate.rb:25:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:in block in run' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/util/busy.rb:19:inbusy' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:in run' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builtin/call.rb:51:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/providers/virtualbox/action/check_virtualbox.rb:17:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/warden.rb:34:in call' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/builder.rb:116:incall' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:in block in run' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/util/busy.rb:19:inbusy' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/action/runner.rb:61:in run' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/machine.rb:147:inaction' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/commands/reload/command.rb:29:in block in execute' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/plugin/v2/command.rb:182:inblock in with_target_vms' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/plugin/v2/command.rb:180:in each' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/plugin/v2/command.rb:180:inwith_target_vms' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/plugins/commands/reload/command.rb:28:in execute' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/cli.rb:46:inexecute' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/lib/vagrant/environment.rb:467:in cli' from /opt/vagrant/embedded/gems/gems/vagrant-1.2.2/bin/vagrant:84:in<top (required)>' from /opt/vagrant/bin/../embedded/gems/bin/vagrant:23:in load' from /opt/vagrant/bin/../embedded/gems/bin/vagrant:23:in

'

[mitchellh]

@arnaudlawson Does this happen every time? This is coming out the Ruby standard library...

[arnaudlawson]

Yes it is happening every time. It doesn't seem to be a ruby issue. This problem only started after we upgraded from Vagrant 1.1.0 to 1.2.2. We even tried to downgrade back to Vagrant 1.1.0 but then encountered the same issue still. So now we are back on Vagrant 1.2.2 and that is running on CentOS release 5.7 (Final) with VirtualBox 4.2.16r86992. Every time we do 'vagrant up' the problem comes up - and we have several boxes that use the second virtual NIC (eth1) to talk to each other. So any help would be appreciated. Thanks.

[arnaudlawson]

@mitchellh

Any thoughts?

[arnaudlawson]

@mitchellh

I found the problem, this was because the /tmp directory was full on this server. "A directory can have at most 31998 subdirectories" while using ext3 filesystems. http://en.wikipedia.org/wiki/Ext3#cite_note-0

Vagrant uses the /tmp directory to build boxes so as i cleaned up /tmp i was able to rebuild the box.

[akoerner]

I'm unfortunately seeing this error again after updating form 1.8.1 to 1.8.4. Host: Win7 Enterprise Vagrant: 1.8.4 Chef DK: chefdk-0.13.21-1-x86 Virtualbox: Version 5.0.20 r106931 vagrant plugin list: vagrant-berkshelf (4.1.0) vagrant-omnibus (1.4.1) vagrant-share (1.1.5, system)

Vagrantfile: http://pastie.org/10877872 vagrant up output: http://pastie.org/10877969