Provisioning node fails -- no node attributes or runlist is found

[jaypipes]

Hi Chris,

Something that was working a couple days ago now seems to be broken. I did a fresh berks update && vagabond server destroy && vagabond server up and the Chef server comes up with no problems.

However, when I go to provision a node, I get failures:

jpipes@uberbox:~/repos/att-cloud/chef-repo$ bundle exec vagabond provision ops
Ensuring expected system state (creating required base containers)
   - This can take a while on first run or new templates...
  -> COMPLETE!
Vagabond: Provisioning node: ops
Bootstrapping Chef on 10.0.3.11
10.0.3.11 Starting Chef Client, version 11.4.4
10.0.3.11 resolving cookbooks for run list: ["ohai", "apt", "bash", "chef-client-wrapper", "sudo", "motd-tail", "self_ca", "dns", "package_installer", "ipmitool", "git", "lldpd", "rsync", "screen", "sysctl", "sysstat", "vim", "ntp", "tmux", "infra-graphing::client", "acct", "tcpspy", "infra-caching", "infra-messaging", "infra-db::metering-master-db"]
10.0.3.11 Synchronizing Cookbooks:
10.0.3.11   - postgresql
10.0.3.11   - mysql
10.0.3.11   - openstack-common
10.0.3.11   - openstack-identity
10.0.3.11   - openstack-network
10.0.3.11   - apache2
10.0.3.11   - openstack-dashboard
10.0.3.11   - selinux
10.0.3.11   - openstack-image
10.0.3.11   - apt
10.0.3.11   - openstack-block-storage
10.0.3.11   - yum
10.0.3.11   - openstack-compute
10.0.3.11   - database
10.0.3.11   - infra-db
10.0.3.11   - build-essential
10.0.3.11   - erlang
10.0.3.11   - rabbitmq
10.0.3.11   - infra-messaging
10.0.3.11   - runit
10.0.3.11   - memcached
10.0.3.11   - infra-caching
10.0.3.11   - tcpspy
10.0.3.11   - acct
10.0.3.11   - ark
10.0.3.11   - passenger_apache2
10.0.3.11   - python
10.0.3.11   - gdash
10.0.3.11   - git
10.0.3.11   - statsd
10.0.3.11   - libapache2-mod-auth-cas
10.0.3.11   - xfs
10.0.3.11   - aws
10.0.3.11   - openssl
10.0.3.11   - sysctl
10.0.3.11   - graphite
10.0.3.11   - diamond
10.0.3.11   - infra-graphing
10.0.3.11   - tmux
10.0.3.11   - ntp
10.0.3.11   - vim
10.0.3.11   - sysstat
10.0.3.11   - screen
10.0.3.11   - rsync
10.0.3.11   - chef_handler
10.0.3.11   - windows
10.0.3.11   - dmg
10.0.3.11   - package_installer
10.0.3.11   - dynect
10.0.3.11   - dns
10.0.3.11   - self_ca
10.0.3.11   - motd-tail
10.0.3.11   - sudo
10.0.3.11   - cron
10.0.3.11   - chef-client
10.0.3.11   - chef-client-wrapper
10.0.3.11   - bash
10.0.3.11   - lldpd
10.0.3.11   - ipmitool
10.0.3.11   - ohai
10.0.3.11 Compiling Cookbooks...
10.0.3.11 /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/provider/lwrp_base.rb:97: warning: already initialized constant AptPreference
10.0.3.11 /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/provider/lwrp_base.rb:97: warning: already initialized constant AptRepository
10.0.3.11 /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/provider/lwrp_base.rb:97: warning: already initialized constant SysctlParam
10.0.3.11 Recipe: ohai::default
10.0.3.11   * remote_directory[/etc/chef/ohai_plugins] action create
10.0.3.11 Recipe: <Dynamically Defined Resource>
10.0.3.11   * cookbook_file[/etc/chef/ohai_plugins/network_ports.rb] action create
10.0.3.11  (up to date)
10.0.3.11   * cookbook_file[/etc/chef/ohai_plugins/network_interface_roles.rb] action create
10.0.3.11  (up to date)
10.0.3.11   * cookbook_file[/etc/chef/ohai_plugins/network_addr.rb] action create
10.0.3.11  (up to date)
10.0.3.11   * cookbook_file[/etc/chef/ohai_plugins/lldp.rb] action create
10.0.3.11  (up to date)
10.0.3.11   * cookbook_file[/etc/chef/ohai_plugins/ipmi.rb] action create
10.0.3.11  (up to date)
10.0.3.11   * cookbook_file[/etc/chef/ohai_plugins/README] action create
10.0.3.11  (up to date)
10.0.3.11  (up to date)
10.0.3.11 
10.0.3.11 Recipe: ohai::default
10.0.3.11   * ohai[custom_plugins] action reload
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Ohai llpd plugin failed with: 'command lldpctl -f keyvalue doesn't exist or is not in the PATH'
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Ohai ipmi plugin failed with: 'command ipmitool lan print doesn't exist or is not in the PATH'
10.0.3.11 
10.0.3.11     - re-run ohai and merge results into node attributes
10.0.3.11 
10.0.3.11 
10.0.3.11 Recipe: chef-client-wrapper::default
10.0.3.11   * template[/usr/local/bin/chef-client] action create
10.0.3.11  (up to date)
10.0.3.11 No packages found matching diamond.
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Cloning resource attributes for service[diamond] from prior resource (CHEF-3694)
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Previous service[diamond]: /var/chef/cache/cookbooks/diamond/recipes/install.rb:78:in `from_file'
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Current  service[diamond]: /var/chef/cache/cookbooks/diamond/recipes/default.rb:5:in `from_file'
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Cloning resource attributes for rabbitmq_user[guest] from prior resource (CHEF-3694)
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Previous rabbitmq_user[guest]: /var/chef/cache/cookbooks/infra-messaging/recipes/default.rb:41:in `from_file'
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Current  rabbitmq_user[guest]: /var/chef/cache/cookbooks/infra-messaging/recipes/default.rb:49:in `from_file'
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Cloning resource attributes for rabbitmq_user[guest] from prior resource (CHEF-3694)
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Previous rabbitmq_user[guest]: /var/chef/cache/cookbooks/infra-messaging/recipes/default.rb:49:in `from_file'
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Current  rabbitmq_user[guest]: /var/chef/cache/cookbooks/infra-messaging/recipes/default.rb:55:in `from_file'
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Cloning resource attributes for rabbitmq_user[guest] from prior resource (CHEF-3694)
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Previous rabbitmq_user[guest]: /var/chef/cache/cookbooks/infra-messaging/recipes/default.rb:55:in `from_file'
10.0.3.11 [2013-07-16T16:07:51+00:00] WARN: Current  rabbitmq_user[guest]: /var/chef/cache/cookbooks/infra-messaging/recipes/default.rb:65:in `from_file'
10.0.3.11 
10.0.3.11 ================================================================================
10.0.3.11 Recipe Compile Error in /var/chef/cache/cookbooks/infra-db/recipes/metering-master-db.rb
10.0.3.11 ================================================================================
10.0.3.11 
10.0.3.11 
10.0.3.11 NoMethodError
10.0.3.11 -------------
10.0.3.11 undefined method `[]' for nil:NilClass
10.0.3.11 
10.0.3.11 
10.0.3.11 
10.0.3.11 Cookbook Trace:
10.0.3.11 ---------------
10.0.3.11   /var/chef/cache/cookbooks/openstack-common/libraries/network.rb:27:in `address_for'
10.0.3.11   /var/chef/cache/cookbooks/infra-db/recipes/metering-db.rb:24:in `from_file'
10.0.3.11   /var/chef/cache/cookbooks/infra-db/recipes/metering-master-db.rb:23:in `from_file'
10.0.3.11 
10.0.3.11 
10.0.3.11 
10.0.3.11 Relevant File Content:
10.0.3.11 ----------------------
10.0.3.11 /var/chef/cache/cookbooks/openstack-common/libraries/network.rb:
10.0.3.11 
10.0.3.11  20:  module ::Openstack
10.0.3.11 
10.0.3.11  21:    # return the IPv4 (default) address of the given interface.
10.0.3.11  22:    #
10.0.3.11 
10.0.3.11  23:    # @param [String] interface The interface to query.
10.0.3.11  24:    # @param [String] family The protocol family to use.
10.0.3.11  25:    # @return [String] The IPv4 address.
10.0.3.11  26:    def address_for interface, family="inet"
10.0.3.11  27>>     interface_node = node["network"]["interfaces"][interface]["addresses"]
10.0.3.11 
10.0.3.11  28:      interface_node.select do |address, data|
10.0.3.11  29:        if data['family'] == family
10.0.3.11 
10.0.3.11  30:          return address
10.0.3.11  31:        end
10.0.3.11  32:      end
10.0.3.11  33:    end
10.0.3.11  34:  end
10.0.3.11 
10.0.3.11  35:  
10.0.3.11 
10.0.3.11 
10.0.3.11 [2013-07-16T16:07:51+00:00] ERROR: Running exception handlers
10.0.3.11 [2013-07-16T16:07:51+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
10.0.3.11 [2013-07-16T16:07:51+00:00] ERROR: Exception handlers complete
10.0.3.11 Chef Client failed. 1 resources updated
10.0.3.11 [2013-07-16T16:07:51+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
10.0.3.11 [2013-07-16T16:07:51+00:00] FATAL: NoMethodError: undefined method `[]' for nil:NilClass
  -> PROVISION FAILED

SSH'ing into the node, I see that the node apparently has no run list, which is odd, since the above shows that indeed the ops node's run list was populated correctly...

jpipes@uberbox:~/repos/att-cloud/chef-repo$ bundle exec vagabond ssh ops
Vagabond: SSH connect to: ops
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.8.0-23-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
Last login: Tue Jul 16 16:07:48 2013 from 10.0.3.1
root@ubuntu1204-cK1VERUbs0Xv:~# chef-client
Starting Chef Client, version 11.4.4
resolving cookbooks for run list: []
Synchronizing Cookbooks:
Compiling Cookbooks...
[2013-07-16T16:08:02+00:00] WARN: Node ops has an empty run list.
Converging 0 resources
Chef Client finished, 0 resources updated

Looking at the node via vagabond knife node show, I see no attributes at all:

jpipes@uberbox:~/repos/att-cloud/chef-repo$ bundle exec vagabond knife node show ops
Node Name:   ops
Environment: vagabond
FQDN:        ubuntu1204-cK1VERUbs0Xv
IP:          10.0.3.11
Run List:    
Roles:       
Recipes:     
Platform:    ubuntu 12.04
Tags:        

Any ideas what might have happened or how I can diagnose? Thanks in advance! -jay

[jaypipes]

Oh... I'm using Berkshelf 2.0.7 and Vagabond your develop branch at SHA1 211e9e11dc817698f139d905b0c228ee3cc563f5

[chrisroberts]

So the node won't have a run list until the converge is successful, at which point the node data is saved, and the run list is populated (so you can see it on the knife node show). Until you get a successful converge, all you'll get is just the basic information you're currently seeing.

This line is causing the problem in the chef run itself:

interface_node = node["network"]["interfaces"][interface]["addresses"]

It looks like what ever interface you're looking for (the value of interface) was not found by ohai.

[jaypipes]

Yeah, this is a problem in a dependent cookbook apparently. Running outside of Vagabond I get the exact same errors... sorry for the noise.