Closed ckurka closed 7 years ago
Hi @ckurka, thanks for the detailed description and environment info. From what I can make of the logs, it looks like it failed inside this block, I'm guessing on the REST POST to start the build plan job. I've seen read timeouts from ICSP before when we try to get this list of available Build Plans; if there are quite a few in there it can take more time than the default 10 seconds given before timeout. In certain cases I've heard of it needing to be 60 seconds or more. ICSP isn't great about sending timely responses in some cases, so let's try bumping up your timeout value to see if that fixes your issue.
You can set the timeout value in your knife.rb file with the knife[:oneview_timeout] = 60
setting; see here for more info.
If that doesn't work, can you please update your description with a little more context? It's helpful to see the log output just before it fails, that way I know where it was in the process.
Thanks for the info I'll give it a try Here it the rest of the log output, as you can see it does return the OS Build Plan, and even starts it (The BP runs to completion in ICSP)
[root@chef examples]# chef-client -z -o provisioning_cookbook [2017-01-13T08:51:19-05:00] INFO: Started chef-zero at chefzero://localhost:8889 with repository at /root/chef-provisioning-oneview/examples One version per cookbook
[2017-01-13T08:51:19-05:00] INFO: Forking chef instance to converge... Starting Chef Client, version 12.17.44 [2017-01-13T08:51:19-05:00] INFO: Chef 12.17.44 [2017-01-13T08:51:19-05:00] INFO: Platform: x86_64-linux [2017-01-13T08:51:19-05:00] INFO: Chef-client pid: 30655 [2017-01-13T08:51:28-05:00] WARN: Run List override has been provided. [2017-01-13T08:51:28-05:00] WARN: Original Run List: [] [2017-01-13T08:51:28-05:00] WARN: Overridden Run List: [recipe[provisioning_cookbook]] [2017-01-13T08:51:28-05:00] INFO: Run List is [recipe[provisioning_cookbook]] [2017-01-13T08:51:28-05:00] INFO: Run List expands to [provisioning_cookbook] [2017-01-13T08:51:28-05:00] INFO: Starting Chef Run for admin [2017-01-13T08:51:28-05:00] INFO: Running start handlers [2017-01-13T08:51:28-05:00] INFO: Start handlers complete. [2017-01-13T08:51:28-05:00] INFO: HTTP Request Returned 404 Not Found: Object not found: resolving cookbooks for run list: ["provisioning_cookbook"] [2017-01-13T08:51:28-05:00] INFO: Loading cookbooks [provisioning_cookbook@0.1.0] [2017-01-13T08:51:28-05:00] INFO: Skipping removal of obsoleted cookbooks from the cache Synchronizing Cookbooks:
provisioning_cookbook (0.1.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 3 resources Recipe: provisioning_cookbook::default
log[Using 'admin' for knife client name] action write[2017-01-13T08:51:28-05:00] INFO: Processing log[Using 'admin' for knife client name] action write (provisioning_cookbook::default line 49) [2017-01-13T08:51:28-05:00] INFO: Using 'admin' for knife client name
log[Using '/root/chef-provisioning-oneview/examples/.chef/admin.pem' for knife client key] action write[2017-01-13T08:51:28-05:00] INFO: Processing log[Using '/root/chef-provisioning-oneview/examples/.chef/admin.pem' for knife client key] action write (provisioning_cookbook::default line 50) [2017-01-13T08:51:28-05:00] INFO: Using '/root/chef-provisioning-oneview/examples/.chef/admin.pem' for knife client key
machine_batch[oneview-machine-batch] action converge[2017-01-13T08:51:28-05:00] INFO: Processing machine_batch[oneview-machine-batch] action converge (provisioning_cookbook::default line 53) [2017-01-13T08:51:28-05:00] INFO: HTTP Request Returned 404 Object Not Found: error
converge
on resource 'machine_batch[oneview-machine-batch]'@jsmartt That got me a LOT further along. Increasing the time out got me past the portion, then I hit an issue with the Network Personalization, I'll go back and double check my knife.rb file, could be a fat finger...
.................................................................
- [chef-web01] Run: ["chef - RHEL 6.6 x64 Scripted Install"] OS Build Plan(s) on chef-web01
- [chef-web01] INFO: Configuring networking on chef-web01
================================================================================
Error executing action `converge` on resource 'machine_batch[oneview-machine-batch]'
================================================================================
RuntimeError
------------
Failed to start network personalization job. Details: The field named bootstrap near line 1, column 228 is not a valid field in this JSON object.
Resource Declaration:
---------------------
# In /root/chef-provisioning-oneview/examples/.chef/local-mode-cache/cache/cookbooks/provisioning_cookbook/recipes/default.rb
53: machine_batch 'oneview-machine-batch' do
54: # Note: Enclosing the machine resources in this machine_batch block allows them to provision in parallel.
55:
56: action action_verb
57:
58: my_machines.each do |m_name, options|
59:
60: machine m_name do
61: recipe 'my_apache_server'
62:
63: machine_options :driver_options => {
64: :server_template => my_template,
65: :os_build => os_build,
66: :host_name => m_name,
67: # :ip_address => options['ip4'], # For bootstrapping. Deprecated in favor of { bootstrap: true } in connection
68:
69: :domainType => 'workgroup',
70: :domainName => domain_name,
71: :gateway => gateway,
72: :mask => mask,
73: :dns => dns,
74: :connections => {
75: # 1 => { dhcp: true }, # (Reserved for PXE)
76: 2 => {
77: :ip4Address => options['ip4'],
78: :dhcp => options['dhcp'] || false,
79: :bootstrap => true
80: }
81: }
82: },
83: :custom_attributes => {},
84: :transport_options => {
85: :ssh_options => {
86: :password => Chef::Config.knife[:node_root_password],
87: :auth_methods => ['password', 'publickey'],
88: :keys => Chef::Config.knife[:node_root_ssh_keys] || []
89: }
90: },
91: :convergence_options => {
92: :ssl_verify_mode => :verify_none,
93: :bootstrap_proxy => Chef::Config.knife[:bootstrap_proxy]
94: }
95:
96: chef_environment '_default'
97: converge true
98: end # End machine resource block
99:
100: end # End my_machines.each loop
101:
102: end # End machine_batch resource block
Compiled Resource:
------------------
# Declared in /root/chef-provisioning-oneview/examples/.chef/local-mode-cache/cache/cookbooks/provisioning_cookbook/recipes/default.rb:53:in `from_file'
machine_batch("oneview-machine-batch") do
action [:converge]
retries 0
retry_delay 2
default_guard_interpreter :default
driver "oneview"
chef_server {:chef_server_url=>"https://chef.atlpss.hp.net/organizations/hpe", :options=>{:client_name=>"admin", :signing_key_filename=>"/root/chef-provisioning-oneview/examples/.chef/admin.pem", :api_version=>"0"}}
declared_type :machine_batch
cookbook_name "provisioning_cookbook"
recipe_name "default"
machines ["chef-web01"]
end
Platform:
---------
x86_64-linux
[2017-01-13T13:22:58-05:00] INFO: Running queued delayed notifications before re-raising exception
Running handlers:
[2017-01-13T13:22:58-05:00] ERROR: Running exception handlers
Running handlers complete
[2017-01-13T13:22:58-05:00] ERROR: Exception handlers complete
Chef Client failed. 2 resources updated in 47 minutes 40 seconds
[2017-01-13T13:22:58-05:00] FATAL: Stacktrace dumped to /root/chef-provisioning-oneview/examples/.chef/local-mode-cache/cache/chef-stacktrace.out
[2017-01-13T13:22:58-05:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2017-01-13T13:22:58-05:00] ERROR: machine_batch[oneview-machine-batch] (provisioning_cookbook::default line 53) had an error: RuntimeError: Failed to start network personalization job. Details: The field named bootstrap near line 1, column 228 is not a valid field in this JSON object.
[2017-01-13T13:22:58-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
TRACE
Generated at 2017-01-13 13:22:58 -0500
RuntimeError: machine_batch[oneview-machine-batch] (provisioning_cookbook::default line 53) had an error: RuntimeError: Failed to start network personalization job. Details: The field named bootstrap near line 1, column 228 is not a valid field in this JSON object.
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-oneview-1.2.0/lib/chef/provisioning/icsp/icsp_api.rb:172:in `block in icsp_configure_networking'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/mixin/why_run.rb:52:in `add_action'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/provider.rb:176:in `converge_by'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-1.9.1/lib/chef/provisioning/chef_provider_action_handler.rb:54:in `perform_action'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-1.9.1/lib/chef/provisioning/add_prefix_action_handler.rb:31:in `perform_action'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-oneview-1.2.0/lib/chef/provisioning/icsp/icsp_api.rb:158:in `icsp_configure_networking'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-oneview-1.2.0/lib/chef/provisioning/customize_machine.rb:37:in `customize_machine'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-oneview-1.2.0/lib/chef/provisioning/oneview_driver.rb:114:in `ready_machine'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-1.9.1/lib/chef/provisioning/driver.rb:250:in `block in ready_machines'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in `call'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in `process_input'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:253:in `process_one'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer.rb:93:in `call'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer.rb:93:in `worker_loop'
>>>> Caused by RuntimeError: Failed to start network personalization job. Details: The field named bootstrap near line 1, column 228 is not a valid field in this JSON object.
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-oneview-1.2.0/lib/chef/provisioning/icsp/icsp_api.rb:172:in `block in icsp_configure_networking'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/mixin/why_run.rb:52:in `add_action'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/provider.rb:176:in `converge_by'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-1.9.1/lib/chef/provisioning/chef_provider_action_handler.rb:54:in `perform_action'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-1.9.1/lib/chef/provisioning/add_prefix_action_handler.rb:31:in `perform_action'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-oneview-1.2.0/lib/chef/provisioning/icsp/icsp_api.rb:158:in `icsp_configure_networking'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-oneview-1.2.0/lib/chef/provisioning/customize_machine.rb:37:in `customize_machine'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-oneview-1.2.0/lib/chef/provisioning/oneview_driver.rb:114:in `ready_machine'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-provisioning-1.9.1/lib/chef/provisioning/driver.rb:250:in `block in ready_machines'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in `call'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in `process_input'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:253:in `process_one'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer.rb:93:in `call'
/usr/local/rvm/gems/ruby-2.2.6/gems/chef-12.17.44/lib/chef/chef_fs/parallelizer.rb:93:in `wor
This is actually a bug @ckurka. I fixed it with PR #53, and you should be able to download the new version of the gem, 1.2.1 with the fix.
@jsmartt Thanks the new version worked like a charm!
First off sorry is this is not the correct spot for this and I this is my first foray into Chef and ruby.
I was able to get everything installed but, I am hitting an issue during the run build plan portion. The build plan executes and runs ,but chef bombs out with a Net::ReadTimeout. (Not sure where to start debugging this)
Thanks -Chris
OV Version is 3.0 ICSP Version is 7.5.1 Chef Version 12.10.0.1 Ruby Version 2.2.6p396 OS Centos 7.0
Here is the error along with the trace