Provisioning recipe errors when running chef-client as a service

[robcoward]

We are just starting out with chef provisioning, but are using a dedicated 'provisioning node' that retrieves a provisioning recipe from a chef server, to then use chef-provisioning-aws to manage elements of our infrastructure in AWS. Converging the cookbook manually on the provisioning node is working fine and all looks good.

However we also use the chef-client::service recipe to setup chef-client as a daemon process. When the chef-client daemon runs the converge it errors at the point that I believe it is trying to read the ~/.aws/credentials file withing the aws driver. The /var/log/chef/client.log file logs the following:

[2015-07-29T10:54:36+01:00] INFO: Processing Cloud3 in developeraws vpc[vpc-57bcb535]
[2015-07-29T10:54:36+01:00] ERROR: Running exception handlers
[2015-07-29T10:54:36+01:00] ERROR: Exception handlers complete
[2015-07-29T10:54:36+01:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-07-29T10:54:36+01:00] INFO: Sending resource update report (run-id: ed4a18ad-2bec-41ab-98f4-2079d7b1397e)
[2015-07-29T10:54:37+01:00] ERROR: undefined method `[]' for nil:NilClass
[2015-07-29T10:54:37+01:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

The stacktrace it generates is:

Generated at 2015-07-29 10:54:36 +0100
NoMethodError: undefined method `[]' for nil:NilClass
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/driver.rb:58:in `initialize'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/driver.rb:46:in `new'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/driver.rb:46:in `from_url'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-1.2.1/lib/chef/provisioning.rb:83:in `driver_for_url'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-1.2.1/lib/chef/provisioning/chef_run_data.rb:105:in `driver_for_url'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-1.2.1/lib/chef/provisioning/chef_run_data.rb:84:in `driver_for'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/aws_resource.rb:46:in `block in <class:AWSResource>'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/super_lwrp.rb:15:in `instance_exec'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/super_lwrp.rb:15:in `block in attribute'
/.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/aws_resource.rb:15:in `initialize'
/opt/chefdk/embedded/apps/chef/lib/chef/resource_builder.rb:48:in `new'
/opt/chefdk/embedded/apps/chef/lib/chef/resource_builder.rb:48:in `build'
/opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:144:in `build_resource'
/opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:105:in `declare_resource'
/opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:44:in `method_missing'
/var/chef/cache/cookbooks/nvm_provisioning/definitions/aws_provisioner_node.rb:12:in `block in from_file'
/opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:74:in `instance_eval'
/opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:74:in `evaluate_resource_definition'
/opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:41:in `method_missing'
/var/chef/cache/cookbooks/nvm_provisioning/recipes/master_provisioner.rb:39:in `block in from_file'
/var/chef/cache/cookbooks/nvm_provisioning/recipes/master_provisioner.rb:23:in `each'
/var/chef/cache/cookbooks/nvm_provisioning/recipes/master_provisioner.rb:23:in `from_file'
/opt/chefdk/embedded/apps/chef/lib/chef/mixin/from_file.rb:30:in `instance_eval'
/opt/chefdk/embedded/apps/chef/lib/chef/mixin/from_file.rb:30:in `from_file'
/opt/chefdk/embedded/apps/chef/lib/chef/cookbook_version.rb:228:in `load_recipe'
/opt/chefdk/embedded/apps/chef/lib/chef/run_context.rb:173:in `load_recipe'
/opt/chefdk/embedded/apps/chef/lib/chef/run_context/cookbook_compiler.rb:140:in `block in compile_recipes'
/opt/chefdk/embedded/apps/chef/lib/chef/run_context/cookbook_compiler.rb:138:in `each'
/opt/chefdk/embedded/apps/chef/lib/chef/run_context/cookbook_compiler.rb:138:in `compile_recipes'
/opt/chefdk/embedded/apps/chef/lib/chef/run_context/cookbook_compiler.rb:75:in `compile'
/opt/chefdk/embedded/apps/chef/lib/chef/run_context.rb:96:in `load'
/opt/chefdk/embedded/apps/chef/lib/chef/policy_builder/expand_node_object.rb:87:in `setup_run_context'
/opt/chefdk/embedded/apps/chef/lib/chef/client.rb:256:in `setup_run_context'
/opt/chefdk/embedded/apps/chef/lib/chef/client.rb:454:in `run'
/opt/chefdk/embedded/apps/chef/lib/chef/application.rb:271:in `block in fork_chef_client'
/opt/chefdk/embedded/apps/chef/lib/chef/application.rb:259:in `fork'
/opt/chefdk/embedded/apps/chef/lib/chef/application.rb:259:in `fork_chef_client'
/opt/chefdk/embedded/apps/chef/lib/chef/application.rb:225:in `block in run_chef_client'
/opt/chefdk/embedded/apps/chef/lib/chef/local_mode.rb:39:in `with_server_connectivity'
/opt/chefdk/embedded/apps/chef/lib/chef/application.rb:213:in `run_chef_client'
/opt/chefdk/embedded/apps/chef/lib/chef/application/client.rb:402:in `block in interval_run_chef_client'
/opt/chefdk/embedded/apps/chef/lib/chef/application/client.rb:392:in `loop'
/opt/chefdk/embedded/apps/chef/lib/chef/application/client.rb:392:in `interval_run_chef_client'
/opt/chefdk/embedded/apps/chef/lib/chef/application/client.rb:382:in `run_application'
/opt/chefdk/embedded/apps/chef/lib/chef/application.rb:60:in `run'
/opt/chefdk/embedded/apps/chef/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:51:in `load'
/usr/bin/chef-client:51:in `<main>'

The relevant line of code mentioned in the stack trace happens to be the first use of a chef-provisioning-aws resource after the with_driver line in the recipe:

    with_driver "aws:#{cloud['Account']}:#{cloud['AWS_Region']}"

  aws_security_group "#{cloud['id']}-provisioning-sg" do
    vpc cloud['vpc_id']
    inbound_rules  [
        {:port => 22..22, :protocol => :tcp, :sources => node['Provisioning']['Subnets']['Office'] },
        {:port => 22..22, :protocol => :tcp, :sources => cloud['vpc_cidr'] }
    ]
  end

Has anyone else had issues using chef-provisioning-aws when running chef-client as a service ?

[stuartpreston]

@robcoward I'm not too familiar with the setup (or the AWS driver specifically), but looking at your stacktrace it does seem as though the credentials hash is not populated before it is used to configure the connection through to AWS. This would as you point out indicate a problem finding the credentials.

The line that sets the path to the credentials file looks like this:

credentials_file = ENV['AWS_CREDENTIAL_FILE'] || File.expand_path('~/.aws/credentials')

So, a couple of questions whilst people familiar with the setup can reply - what user is the daemon process set to run under? Does that user have a login that sets $HOME correctly? Are you using stop-start-daemon. Can you set the AWS_CREDENTIAL_FILE and AWS_CONFIG_FILE environment variables? Also what distribution are you using?

[christinedraper]

It looks somewhat similar to:

https://github.com/chef/chef-provisioning-aws/issues/260

which was caused by a node having an old format driver url stored in its attributes. In your case, it might be that an old driver url has been stored for the security group in the aws_security_group data bag

Regards, Christine

On Wed, Jul 29, 2015 at 5:29 AM, Rob Coward notifications@github.com wrote:

We are just starting out with chef provisioning, but are using a dedicated 'provisioning node' that retrieves a provisioning recipe from a chef server, to then use chef-provisioning-aws to manage elements of our infrastructure in AWS. Converging the cookbook on the provisioning node is working fine and all looks good.

However we also use the chef-client::service recipe to setup chef-client as a daemon process. When the chef-client daemon runs the converge it errors at the point that I believe it is trying to read the ~/.aws/credentials file withing the aws driver. The /var/log/chef/client.log file logs the following:

[2015-07-29T10:54:36+01:00] INFO: Processing Cloud3 in developeraws vpc[vpc-57bcb535] [2015-07-29T10:54:36+01:00] ERROR: Running exception handlers [2015-07-29T10:54:36+01:00] ERROR: Exception handlers complete [2015-07-29T10:54:36+01:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out [2015-07-29T10:54:36+01:00] INFO: Sending resource update report (run-id: ed4a18ad-2bec-41ab-98f4-2079d7b1397e) [2015-07-29T10:54:37+01:00] ERROR: undefined method `[]' for nil:NilClass [2015-07-29T10:54:37+01:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

The stacktrace it generates is:

Generated at 2015-07-29 10:54:36 +0100 NoMethodError: undefined method []' for nil:NilClass /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/driver.rb:58:ininitialize' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/driver.rb:46:in new' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/driver.rb:46:infrom_url' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-1.2.1/lib/chef/provisioning.rb:83:in driver_for_url' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-1.2.1/lib/chef/provisioning/chef_run_data.rb:105:indriver_for_url' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-1.2.1/lib/chef/provisioning/chef_run_data.rb:84:in driver_for' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/aws_resource.rb:46:inblock in class:AWSResource' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/super_lwrp.rb:15:in instance_exec' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/super_lwrp.rb:15:inblock in attribute' /.chefdk/gem/ruby/2.1.0/gems/chef-provisioning-aws-1.3.0/lib/chef/provisioning/aws_driver/aws_resource.rb:15:in initialize' /opt/chefdk/embedded/apps/chef/lib/chef/resource_builder.rb:48:innew' /opt/chefdk/embedded/apps/chef/lib/chef/resource_builder.rb:48:in build' /opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:144:inbuild_resource' /opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:105:in declare_resource' /opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:44:inmethod_missing' /var/chef/cache/cookbooks/nvm_provisioning/definitions/aws_provisioner_node.rb:12:in block in from_file' /opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:74:ininstance_eval' /opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:74:in evaluate_resource_definition' /opt/chefdk/embedded/apps/chef/lib/chef/dsl/recipe.rb:41:inmethod_missing' /var/chef/cache/cookbooks/nvm_provisioning/recipes/master_provisioner.rb:39:in block in from_file' /var/chef/cache/cookbooks/nvm_provisioning/recipes/master_provisioner.rb:23:ineach' /var/chef/cache/cookbooks/nvm_provisioning/recipes/master_provisioner.rb:23:in from_file' /opt/chefdk/embedded/apps/chef/lib/chef/mixin/from_file.rb:30:ininstance_eval' /opt/chefdk/embedded/apps/chef/lib/chef/mixin/from_file.rb:30:in from_file' /opt/chefdk/embedded/apps/chef/lib/chef/cookbook_version.rb:228:inload_recipe' /opt/chefdk/embedded/apps/chef/lib/chef/run_context.rb:173:in load_recipe' /opt/chefdk/embedded/apps/chef/lib/chef/run_context/cookbook_compiler.rb:140:inblock in compile_recipes' /opt/chefdk/embedded/apps/chef/lib/chef/run_context/cookbook_compiler.rb:138:in each' /opt/chefdk/embedded/apps/chef/lib/chef/run_context/cookbook_compiler.rb:138:incompile_recipes' /opt/chefdk/embedded/apps/chef/lib/chef/run_context/cookbook_compiler.rb:75:in compile' /opt/chefdk/embedded/apps/chef/lib/chef/run_context.rb:96:inload' /opt/chefdk/embedded/apps/chef/lib/chef/policy_builder/expand_node_object.rb:87:in setup_run_context' /opt/chefdk/embedded/apps/chef/lib/chef/client.rb:256:insetup_run_context' /opt/chefdk/embedded/apps/chef/lib/chef/client.rb:454:in run' /opt/chefdk/embedded/apps/chef/lib/chef/application.rb:271:inblock in fork_chef_client' /opt/chefdk/embedded/apps/chef/lib/chef/application.rb:259:in fork' /opt/chefdk/embedded/apps/chef/lib/chef/application.rb:259:infork_chef_client' /opt/chefdk/embedded/apps/chef/lib/chef/application.rb:225:in block in run_chef_client' /opt/chefdk/embedded/apps/chef/lib/chef/local_mode.rb:39:inwith_server_connectivity' /opt/chefdk/embedded/apps/chef/lib/chef/application.rb:213:in run_chef_client' /opt/chefdk/embedded/apps/chef/lib/chef/application/client.rb:402:inblock in interval_run_chef_client' /opt/chefdk/embedded/apps/chef/lib/chef/application/client.rb:392:in loop' /opt/chefdk/embedded/apps/chef/lib/chef/application/client.rb:392:ininterval_run_chef_client' /opt/chefdk/embedded/apps/chef/lib/chef/application/client.rb:382:in run_application' /opt/chefdk/embedded/apps/chef/lib/chef/application.rb:60:inrun' /opt/chefdk/embedded/apps/chef/bin/chef-client:26:in <top (required)>' /usr/bin/chef-client:51:inload' /usr/bin/chef-client:51:in `

'

The relevant line of code mentioned in the stack trace happens to be the first use of a chef-provisioning-aws resource after the with_driver line in the recipe:

with_driver "aws:#{cloud['Account']}:#{cloud['AWS_Region']}"

aws_security_group "#{cloud['id']}-provisioning-sg" do vpc cloud['vpc_id'] inbound_rules [ {:port => 22..22, :protocol => :tcp, :sources => node['Provisioning']['Subnets']['Office'] }, {:port => 22..22, :protocol => :tcp, :sources => cloud['vpc_cidr'] } ] end

Has anyone else had issues using chef-provisioning-aws when running chef-client as a service ?

— Reply to this email directly or view it on GitHub https://github.com/chef/chef-provisioning-aws/issues/266.

ThirdWave Insights, LLC I (512) 971-8727 <%28512%29%20656-7724> I www.ThirdWaveInsights.com I P.O. Box 500134 I Austin, TX 78750

[robcoward]

@christinedraper I dont think its a driver url issue here since chef-client does converge correctly when run manually.

Its more likely to be an environment issue, like @stuartpreston suggests. We are just using the chef-client::service recipe out of the box that sets up the /etc/init.d/chef-client script to run chef-client as the root user (same as when we converge manually). When checking the environment for the PID of the running chef-client service however, the number of environment variables seems rather lacking:

# cat /proc/8910/environ
TERM=xtermPATH=/sbin:/usr/sbin:/bin:/usr/binPWD=/LANG=en_US.UTF-8SHLVL=2_=/usr/bin/chef-client

That probably explains why File.expand_path('~/.aws/credentials') fails to find the credentials.

Does anyone else use chef-provisioning-aws in conjunction with the chef-client cookbook ?