search

CenturyLinkCloud / chef-provisioning-vsphere

A chef-provisioning provisioner for VMware vSphere
MIT License
66 stars 57 forks source link

Long Chef Provisioning runs result in vSphere timeouts #43

Open troyready opened 8 years ago

troyready commented 8 years ago

I just encountered the following exception during a chef-provisioning client run:

    RbVmomi::Fault
    --------------
    NotAuthenticated:

I'm betting this occurred because the first machine convergence took over 45 minutes (if that's a bad assumption disregard the rest of this and let me know).

From what I can tell it should be simple enough to add connection retry support into the first vim conditional in the helper library. Thoughts?

mwrock commented 8 years ago

What was the stack trace of the exception? All calls to vim check to see that there is a currently logged in connection and if not it reauthenticates and creates a new one. So the connection may indeed expire after a long converge but the vim method should renew it.

It would be interesting to see the stack trace in case the connection is coming from somewhere else other than the vim method.

Another thing...if you wrap your machine resources in a machine_batch resource. The convergences will run concurrently after all the machines have been created.

troyready commented 8 years ago

Thanks for the tip about machine_batch, I did not know that. That solves my immediate problem, but for the sake of completeness I recreated the issue today. It's possible that my driver.rb line numbers don't line up exactly with the rubygems.org version because I'm using my changes from #42 , but I'm pretty sure that isn't causing the issue:

Generated at 2015-11-02 15:28:48 -0800
RbVmomi::Fault: machine[secondmachineresource.myorg.com] (myprovisioningcookbook::longtest line 67) had an error: RbVmomi::Fault: NotAuthenticated: 
/home/troyready/.chefdk/gem/ruby/2.1.0/gems/rbvmomi-1.8.2/lib/rbvmomi/basic_types.rb:192:in `_get_property'
/home/troyready/.chefdk/gem/ruby/2.1.0/gems/rbvmomi-1.8.2/lib/rbvmomi/basic_types.rb:35:in `block (2 levels) in init'
/home/troyready/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vsphere-0.8.1/lib/chef/provisioning/vsphere_driver/vsphere_helpers.rb:73:in `find_folder'
/home/troyready/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vsphere-0.8.1/lib/chef/provisioning/vsphere_driver/vsphere_helpers.rb:38:in `find_vm'
/home/troyready/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vsphere-0.8.1/lib/chef/provisioning/vsphere_driver/driver.rb:166:in `find_or_create_vm'
/home/troyready/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-vsphere-0.8.1/lib/chef/provisioning/vsphere_driver/driver.rb:135:in `allocate_machine'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-1.4.1/lib/chef/provider/machine.rb:31:in `block in <class:Machine>'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:362:in `action_allocate'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-1.4.1/lib/chef/provider/machine.rb:38:in `block in <class:Machine>'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:362:in `action_ready'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-1.4.1/lib/chef/provider/machine.rb:56:in `block in <class:Machine>'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:362:in `action_converge'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:144:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource.rb:585:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:49:in `run_action'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `each'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `block in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:83:in `block in execute_each_resource'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:81:in `execute_each_resource'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:80:in `converge'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:653:in `block in converge'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:648:in `catch'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:648:in `converge'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:687:in `converge_and_save'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:269:in `run'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:270:in `block in fork_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:258:in `fork'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:258:in `fork_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:224:in `block in run_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:212:in `run_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:408:in `block in interval_run_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:398:in `loop'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:398:in `interval_run_chef_client'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:388:in `run_application'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:60:in `run'
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:54:in `load'
/usr/bin/chef-client:54:in `<main>'
mwrock commented 8 years ago

Ah ok. Thats helpful. Looks like the datacenter is cached and using the cached datacenter after the connection expires causes this error. Ideally the SessionManager would be checked to see if the connection was still active before returning the cached datacenter instance. Otherwise, reestablish the connection and return a new datacenter.