Closed irifed closed 10 years ago
ju2wheels
commented
10 years ago Turning it on is pretty straightforward according to the docs, whether softlayer_api (/xmlrpc), and what we have ontop of softlayer_api are thread safe is another. If each box is creating its own SL API service instances then I dont think we have anything to worry about, but im not sure if they are or not.
@underscorephil : Any thoughts on softlayer_api thread safety?
ju2wheels
commented
10 years ago @irifed: To enable until its officially enabled, edit ~/.vagrant.d/gems/gems/vagrant-softlayer-0.3.1/lib/vagrant-softlayer/plugin.rb:
Change:
provider(:softlayer) do
to
provider(:softlayer, :parallel => true) do
Throws a few errors but otherwise works:
$ vagrant up --parallel
ERROR loader: Unknown config sources: ["23729360_machine_tempvagrantbuildparallel1", :"19675120_ju2wheels/SL_UBUNTU_LATEST_64_temp_softlayer", :"23729360_vm_tempvagrantbuildparallel1_ju2wheels/SL_UBUNTU_LATEST_64_temp_softlayer"]
ERROR loader: Unknown config sources: ["23729360_machine_tempvagrantbuildparallel1", :"23729360_vm_tempvagrantbuildparallel1_ju2wheels/SL_UBUNTU_LATEST_64_temp_softlayer"]
ERROR loader: Unknown config sources: ["23729360_machine_tempvagrantbuildparallel1", :"23729360_vm_tempvagrantbuildparallel1_ju2wheels/SL_UBUNTU_LATEST_64_temp_softlayer"]
Bringing machine 'tempvagrantbuildparallel1' up with 'softlayer' provider...
Bringing machine 'tempvagrantbuildparallel' up with 'softlayer' provider...
==> tempvagrantbuildparallel1: Creating a new SoftLayer instance...
==> tempvagrantbuildparallel: Creating a new SoftLayer instance...
==> tempvagrantbuildparallel: Waiting for instance provisioning. This may take a few minutes...
==> tempvagrantbuildparallel1: Waiting for instance provisioning. This may take a few minutes...
==> tempvagrantbuildparallel: SoftLayer instance successfully provisioned!
==> tempvagrantbuildparallel: Waiting for machine to boot. This may take a few minutes...
tempvagrantbuildparallel: SSH address: 10.x.x.x:22
tempvagrantbuildparallel: SSH username: root
tempvagrantbuildparallel: SSH auth method: private key
==> tempvagrantbuildparallel: Machine booted and ready!
==> tempvagrantbuildparallel1: SoftLayer instance successfully provisioned!
==> tempvagrantbuildparallel1: Waiting for machine to boot. This may take a few minutes...
==> tempvagrantbuildparallel: Rsyncing folder: /home/jalajara/IBM/repos/chef/chef-starter/infra/test/ustl0-in00-is25/ => /vagrant
tempvagrantbuildparallel1: SSH address: 10.x.x.x:22
tempvagrantbuildparallel1: SSH username: root
tempvagrantbuildparallel1: SSH auth method: private key
==> tempvagrantbuildparallel1: Machine booted and ready!
==> tempvagrantbuildparallel1: Rsyncing folder: /home/jalajara/IBM/repos/chef/chef-starter/infra/test/ustl0-in00-is25/ => /vagrant
$ vagrant destroy --force
ERROR loader: Unknown config sources: ["14018780_machine_tempvagrantbuildparallel1", :"10089640_ju2wheels/SL_UBUNTU_LATEST_64_temp_softlayer", :"14018780_vm_tempvagrantbuildparallel1_ju2wheels/SL_UBUNTU_LATEST_64_temp_softlayer"]
ERROR loader: Unknown config sources: ["14018780_machine_tempvagrantbuildparallel1", :"14018780_vm_tempvagrantbuildparallel1_ju2wheels/SL_UBUNTU_LATEST_64_temp_softlayer"]
ERROR loader: Unknown config sources: ["14018780_machine_tempvagrantbuildparallel1", :"14018780_vm_tempvagrantbuildparallel1_ju2wheels/SL_UBUNTU_LATEST_64_temp_softlayer"]
==> tempvagrantbuildparallel: Destroying the SoftLayer instance...
==> tempvagrantbuildparallel1: Destroying the SoftLayer instance...[Edit] I had a modified Vagrant log verbosity, when i retried with a normal log verbosity I got no errors so it looks ok.
underscorephil
commented
10 years ago Howdy!
I just spoke with @slsthompson. While there is nothing in the lib to ensure thread safety, he nor I see anything that would prevent the use of the parallel feature.
irifed
commented
10 years ago @ju2wheels : thanks for suggestion!
However, I just tried adding :parallel as you suggested and sometimes (not always) noticed the following error:
$ vagrant up --provider=softlayer --parallel --no-provision
Bringing machine 'host1' up with 'softlayer' provider...
Bringing machine 'host2' up with 'softlayer' provider...
==> host1: HandleBoxUrl middleware is deprecated. Use HandleBox instead.
==> host2: HandleBoxUrl middleware is deprecated. Use HandleBox instead.
==> host2: This is a bug with the provider. Please contact the creator
==> host1: This is a bug with the provider. Please contact the creator
==> host2: of the provider you use to fix this.
==> host1: of the provider you use to fix this.
==> host2: Creating a new SoftLayer instance...
==> host1: Creating a new SoftLayer instance...
==> host1: An error occurred. The error will be shown after all tasks complete.
==> host2: Waiting for instance provisioning. This may take a few minutes...
==> host2: SoftLayer instance successfully provisioned!
==> host2: Waiting for machine to boot. This may take a few minutes...
host2: SSH address: 5.153.56.179:22
host2: SSH username: root
host2: SSH auth method: private key
==> host2: Machine booted and ready!
==> host2: Rsyncing folder: /Users/irina/Projects/cloud/vagrant/softlayer/ => /vagrant
==> host2: Machine not provisioning because `--no-provision` is specified.
An error occurred while executing multiple actions in parallel.
Any errors that occurred are shown below.
An unexpected error occurred when executing the action on the
'host1' machine. Please report this as a bug:
Net::ReadTimeout
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue in rbuf_fill'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/http.rb:1406:in `block in transport_request'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/http.rb:1403:in `catch'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/http.rb:1403:in `transport_request'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/http.rb:1376:in `request'
/Users/irina/.vagrant.d/gems/gems/softlayer_api-1.0.8/lib/softlayer/service.rb:373:in `block in issue_http_request'
/Applications/Vagrant/embedded/lib/ruby/2.0.0/net/http.rb:852:in `start'
/Users/irina/.vagrant.d/gems/gems/softlayer_api-1.0.8/lib/softlayer/service.rb:369:in `issue_http_request'
/Users/irina/.vagrant.d/gems/gems/softlayer_api-1.0.8/lib/softlayer/service.rb:279:in `call_softlayer_api_with_params'
/Users/irina/.vagrant.d/gems/gems/softlayer_api-1.0.8/lib/softlayer/service.rb:246:in `method_missing'
/Users/irina/.vagrant.d/gems/gems/vagrant-softlayer-0.3.1/lib/vagrant-softlayer/action/create_instance.rb:20:in `block in call'
/Users/irina/.vagrant.d/gems/gems/vagrant-softlayer-0.3.1/lib/vagrant-softlayer/util/warden.rb:18:in `sl_warden'
/Users/irina/.vagrant.d/gems/gems/vagrant-softlayer-0.3.1/lib/vagrant-softlayer/action/create_instance.rb:20:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Users/irina/.vagrant.d/gems/gems/vagrant-softlayer-0.3.1/lib/vagrant-softlayer/action/sync_folders.rb:21:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/builtin/provision.rb:80:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Users/irina/.vagrant.d/gems/gems/vagrant-softlayer-0.3.1/lib/vagrant-softlayer/action/setup_softlayer.rb:33:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/builder.rb:116:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/runner.rb:66:in `block in run'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/util/busy.rb:19:in `busy'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/runner.rb:66:in `run'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/builtin/call.rb:53:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Users/irina/.vagrant.d/gems/gems/vagrant-softlayer-0.3.1/lib/vagrant-softlayer/action/setup_softlayer.rb:33:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/builtin/handle_box.rb:56:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/builtin/handle_box_url.rb:9:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/warden.rb:34:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/builder.rb:116:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/runner.rb:66:in `block in run'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/util/busy.rb:19:in `busy'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/action/runner.rb:66:in `run'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/machine.rb:196:in `action_raw'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/machine.rb:173:in `block in action'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/environment.rb:434:in `lock'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/machine.rb:161:in `call'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/machine.rb:161:in `action'
/Applications/Vagrant/embedded/gems/gems/vagrant-1.6.3/lib/vagrant/batch_action.rb:82:in `block (2 levels) in run'Looks like the error is caused by some timeout while waiting for provisioning. I've found a couple of timeout values in the plugin source, but they are set to 20mins. I was getting these errors after ~4mins of waiting. Same Vagrantfile works without errors if parallel is not enabled. Any ideas which timeout should be increased?
Thanks!
ju2wheels
commented
10 years ago The only vagrant specific timeouts im aware of are the boot_timeout and graceful_halt_timeout, see the template at the bottom of the Quick Start Guide.
Based on the above though, I dont think vagrant/vagrant-softlayer is the issue since if we raise those values high enough we will then be bound by the default HTTP timeouts associated with softlayer_api XMLRPC which is where its blowing up anyway and see if raising it helps or if its other network issue.
I tried to issue SoftLayer::Service#xmlrpc_client.timeout= call but its private so I dont think we can increase the timeout currently. SoftLayer::Service The default timeout is 30s according to docs.
[Edit] Was looking at the wrong version but both seem to have the same problem: 1.x branch is based on Net::HTTPS (w/ 60s default read_timeout) and the 2.x on XMLRPC (w/30s default timeout as far as I can tell).
SLsthompson
commented
10 years ago Adding the timeout is a source compatible change. Let me see if I can get that in there and put out a dot release.
marianormuro
commented
10 years ago Hi guys, just wanted to add interest to this thread. Ill be trying the tip for enabling parallel support tonight and would be happy to test any release with official support.
Cheers
ju2wheels
commented
10 years ago @marianormuro : beta testing latest patches without manually changing:
git clone https://github.com/ju2wheels/vagrant-softlayer.git
cd vagrant-softlayer
gem build vagrant-softlayer.gemspec
vagrant plugin uninstall vagrant-softlayer
vagrant plugin install vagrant-softlayer-0.3.1.gemThis will include all changes in https://github.com/audiolize/vagrant-softlayer/pull/15
ju2wheels
commented
10 years ago @irifed : thanks to @SLsthompson for some quick fixes on softlayer_api I think we are ready to beta test a timeout feature to see if it resolves your issue.
Please try the following:
$ vagrant plugin uninstall vagrant-softlayer
$ vagrant plugin uninstall softlayer_api
$ git clone https://github.com/softlayer/softlayer-ruby.git
$ cd softlayer-ruby/
$ git checkout Version_2.2.0
$ gem build softlayer_api.gemspec
$ vagrant plugin install softlayer_api-2.2.0.gem
$ cd ..
$ git clone https://github.com/ju2wheels/vagrant-softlayer.git
$ cd vagrant-softlayer
$ git checkout parallel_with_timeout
$ gem build vagrant-softlayer.gemspec
$ vagrant plugin install vagrant-softlayer-0.3.1.gemWarning: Installing softlayer_api may trigger vagrant to install nokogiri (or at least it did for me, even though I dont think its needed) and the latest version will require that you have Xcode and gcc equivalent compiler if using a Mac I think (theres a bunch of open tickets upstream about this).
Set api_timeout value to change the timeout in the SoftLayer provider.
SLsthompson
commented
10 years ago That seems weird.
softlayer_api by itself doesn't depend on nokogiri an none of the gems we depend on do either. It it something in the vagrant environment that asks you to install nokogiri when the softlayer_api gem is used?
ju2wheels
commented
10 years ago Based on what I have read upstream its something to do with the way Vagrant uses bundler.
SLsthompson
commented
10 years ago OK. Just wanted to make sure that I wasn't doing something silly... wouldn't have been the first time.
emyl
commented
10 years ago @ju2wheels I've just pushed a development branch, please use that for any future pull request. Thanks :wink:
emyl
commented
10 years ago Parallel execution capability has been added with the release 0.3.3
lonniev
commented
10 years ago In 0.3.3, with multiple instance config blocks in a single file and without specifying the --parallel flag, one can get errors in one or more host initializations if the instances contend for common files such as the cookbook rb files. For example, I had 3 out of 4 instances successfully initialize but the 4th failed when it complained that one of the cookbook rb files did not exist. The file did exist, the first 3 found it, and it was present at the end of the run. There may be a file access timeout that the 4th accessor experienced that the first 3 did not.
I'll try with --parallel to see if that makes a difference.
ju2wheels
commented
10 years ago @lonniev can you provide a stripped down example and output of the error? If you arent using the parallel flag then they are going sequentially and wouldnt make sense for their to be contention for file access on the vagrant host unless one of the instances is depending on a file created by a previous one or a file removed by a previous one...
Please open as a separate issue.
lonniev
commented
10 years ago I’ll commit the current Vagrantfile and then open a separate issue for the file contention issue.
—Lonnie VanZandt
303-900-3048 Sent from Dropbox's Mailbox on Mac
On Mon, Oct 27, 2014 at 6:38 PM, Julio Lajara notifications@github.com wrote:
@lonniev can you provide a stripped down example and output of the error? If you arent using the parallel flag then they are going sequentially and wouldnt make sense for their to be contention for file access on the vagrant host unless one of the instances is depending on a file created by a previous one or a file removed by a previous one...
Please open as a separate issue.
Reply to this email directly or view it on GitHub: https://github.com/audiolize/vagrant-softlayer/issues/16#issuecomment-60693657
Hi @emyl , I noticed that currently it is not possible to create instances in parallel with
--parallelflag. Are there any plans to enable this?I would be happy to contribute, but I am new to Vagrant plugins development. Do you think this would take a lot of effort? Where to look at first?
Thanks!