Closed Stromweld closed 7 years ago
tas50
commented
7 years ago I'm a bit confused on where exactly this failure is coming from, but it looks like you're using the aws:ec2_hints recipe. The good news is you don't need that anymore. You can just remove that from your runlist entirely. Chef will autodetect AWS without issue now.
Stromweld
commented
7 years ago So I removed the aws::ec2_hints recipe from our base cookbook, but when our ohai-nativex cookbook tries to create an awsplugin for ohai and sets ['aws']['region'] attribute we get the same error:
[2017-04-19T16:33:59+00:00] WARN: Plugin Network: unable to detect ipaddress
resolving cookbooks for run list: ["role-base-amz-linux-nativex"][0m
Synchronizing Cookbooks:[0m
- chef-sugar (3.4.0)[0m
- nano (1.0.3)[0m
- line (0.6.3)[0m
- motd (0.6.4)[0m
- hostsfile (2.4.5)[0m
- dstat (0.1.1)[0m
- sshd (1.3.0)[0m
- bash-completion (0.1.0)[0m
- compat_resource (12.16.3)[0m
- vim (2.0.2)[0m
- yum-epel (2.1.1)[0m
- logrotate (2.1.0)[0m
- sudo (3.3.1)[0m
- now (1.0.0)[0m
- ssh_authorized_keys (0.3.0)[0m
- snmp (4.0.0)[0m
- nfs (2.3.1)[0m
- seven_zip (2.0.2)[0m
- chef_handler (2.1.1)[0m
- mingw (2.0.0)[0m
- sysctl (0.8.1)[0m
- collectd-ng (2.2.0)[0m
- aws-tagging-nativex (1.1.4)[0m
- ephemeral_lvm-nativex (1.0.6)[0m
- aws (5.0.0)[0m
- ohai (5.0.3)[0m
- lvm (4.0.6)[0m
- chef_client_updater (1.0.2)[0m
- ssh-authorized-keys-nativex (1.0.3)[0m
- ntp-nativex (1.1.3)[0m
- mlocate-nativex (1.0.3)[0m
- windows (3.0.5)[0m
- autofs-nativex (1.1.4)[0m
- iptables (4.1.0)[0m
- ohai-nativex (1.0.5)[0m
- cron (4.1.1)[0m
- ephemeral_lvm (3.0.1)[0m
- graphite-nativex (1.1.6)[0m
- postfix-nativex (1.0.3)[0m
- solarwinds-nativex (1.5.3)[0m
- timezone-ii (0.2.0)[0m
- ad-nativex (3.3.6)[0m
- autopatch-nativex (1.3.6)[0m
- chef-client (8.0.1)[0m
- auto-patch (0.1.10)[0m
- build-essential (8.0.1)[0m
- timezone-nativex (1.0.5)[0m
- hostname-nativex (1.0.8)[0m
- role-base-amz-linux-nativex (1.2.4)[0m
Installing Cookbook Gems:[0m
Compiling Cookbooks...[0m
/var/chef/cache/cookbooks/hostsfile/resources/entry.rb:35: warning: constant ::Fixnum is deprecated
Recipe: ohai-nativex::awsplugin[0m
* ohai_hint[aws] action create
* directory[/etc/chef/ohai/hints] action create (up to date)
* file[/etc/chef/ohai/hints/aws.json] action create
[32m- create new file /etc/chef/ohai/hints/aws.json[0m[0m
================================================================================[0m
[31mError executing action `create` on resource 'file[/etc/chef/ohai/hints/aws.json]'[0m
================================================================================[0m
[0m NoMethodError[0m
-------------[0m
undefined method `split' for nil:NilClass[0m
[0m Cookbook Trace:[0m
---------------[0m
/var/chef/cache/cookbooks/ohai/resources/hint.rb:53:in `block in after_created'
[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `each'
[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `after_created'
[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb:10:in `from_file'
[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/default.rb:10:in `from_file'
[0m /var/chef/cache/cookbooks/role-base-amz-linux-nativex/recipes/default.rb:151:in `from_file'[0m
[0m Resource Declaration:[0m
---------------------[0m
# In /var/chef/cache/cookbooks/ohai/resources/hint.rb
[0m
[0m 11: file ohai_hint_path do
[0m 12: action :create
[0m 13: content build_content
[0m 14: end
[0m 15: end
[0m
[0m Compiled Resource:[0m
------------------[0m
# Declared in /var/chef/cache/cookbooks/ohai/resources/hint.rb:11:in `block in class_from_file'
[0m
[0m file("/etc/chef/ohai/hints/aws.json") do
[0m action [:create]
[0m default_guard_interpreter :default
[0m declared_type :file
[0m cookbook_name "ohai-nativex"
[0m path "/etc/chef/ohai/hints/aws.json"
[0m owner nil
[0m group nil
[0m mode nil
[0m verifications []
[0m end
[0m
[0m System Info:[0m
------------[0m
chef_version=13.0.118
[0m platform=amazon
[0m platform_version=2016.09
[0m ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
[0m program_name=chef-client worker: ppid=32501;start=16:33:58;
[0m executable=/opt/chef/bin/chef-client[0m
[0m [0m
================================================================================[0m
[31mError executing action `create` on resource 'ohai_hint[aws]'[0m
================================================================================[0m
[0m NoMethodError[0m
-------------[0m
file[/etc/chef/ohai/hints/aws.json] (/var/chef/cache/cookbooks/ohai/resources/hint.rb line 11) had an error: NoMethodError: undefined method `split' for nil:NilClass[0m
[0m Cookbook Trace:[0m
---------------[0m
/var/chef/cache/cookbooks/ohai/resources/hint.rb:53:in `block in after_created'
[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `each'
[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `after_created'
[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb:10:in `from_file'
[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/default.rb:10:in `from_file'
[0m /var/chef/cache/cookbooks/role-base-amz-linux-nativex/recipes/default.rb:151:in `from_file'[0m
[0m Resource Declaration:[0m
---------------------[0m
# In /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb
[0m
[0m 10: ohai_hint 'aws'
[0m 11:
[0m
[0m Compiled Resource:[0m
------------------[0m
# Declared in /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb:10:in `from_file'
[0m
[0m ohai_hint("aws") do
[0m action [:create]
[0m default_guard_interpreter :default
[0m declared_type :ohai_hint
[0m cookbook_name "ohai-nativex"
[0m recipe_name "awsplugin"
[0m hint_name "aws"
[0m end
[0m
[0m System Info:[0m
------------[0m
chef_version=13.0.118
[0m platform=amazon
[0m platform_version=2016.09
[0m ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
[0m program_name=chef-client worker: ppid=32501;start=16:33:58;
[0m executable=/opt/chef/bin/chef-client[0m
[0m [0m
================================================================================[0m
[31mRecipe Compile Error in /var/chef/cache/cookbooks/role-base-amz-linux-nativex/recipes/default.rb[0m
================================================================================[0m
[0m NoMethodError[0m
-------------[0m
ohai_hint[aws] (ohai-nativex::awsplugin line 10) had an error: NoMethodError: file[/etc/chef/ohai/hints/aws.json] (/var/chef/cache/cookbooks/ohai/resources/hint.rb line 11) had an error: NoMethodError: undefined method `split' for nil:NilClass[0m
[0m Cookbook Trace:[0m
---------------[0m
/var/chef/cache/cookbooks/ohai/resources/hint.rb:53:in `block in after_created'
[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `each'
[0m /var/chef/cache/cookbooks/ohai/resources/hint.rb:52:in `after_created'
[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/awsplugin.rb:10:in `from_file'
[0m /var/chef/cache/cookbooks/ohai-nativex/recipes/default.rb:10:in `from_file'
[0m /var/chef/cache/cookbooks/role-base-amz-linux-nativex/recipes/default.rb:151:in `from_file'[0m
[0m Relevant File Content:[0m
----------------------[0m
/var/chef/cache/cookbooks/ohai/resources/hint.rb:
[0m
[0m 46: end
[0m 47: end
[0m 48:
[0m 49: # this resource forces itself to run at compile_time
[0m 50: def after_created
[0m 51: return unless compile_time
[0m 52: Array(action).each do |action|
[0m 53>> run_action(action)
[0m 54: end
[0m 55: end
[0m 56: [0m
[0m System Info:[0m
------------[0m
chef_version=13.0.118
[0m platform=amazon
[0m platform_version=2016.09
[0m ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
[0m program_name=chef-client worker: ppid=32501;start=16:33:58;
[0m executable=/opt/chef/bin/chef-client[0m
[0m [0m
Running handlers:[0m
[2017-04-19T16:34:01+00:00] ERROR: Running exception handlers
Running handlers complete
[0m[2017-04-19T16:34:01+00:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 03 seconds[0m
[2017-04-19T16:34:01+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2017-04-19T16:34:01+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2017-04-19T16:34:01+00:00] ERROR: ohai_hint[aws] (ohai-nativex::awsplugin line 10) had an error: NoMethodError: file[/etc/chef/ohai/hints/aws.json] (/var/chef/cache/cookbooks/ohai/resources/hint.rb line 11) had an error: NoMethodError: undefined method `split' for nil:NilClass
[2017-04-19T16:34:01+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1).```Here's our awsplugin recipe:
ohai_hint 'aws'
ohai 'reload_aws' do
plugin 'aws'
action :nothing
end
ohai_plugin 'aws' do
notifies :reload, 'ohai[reload_aws]', :immediately
endand here is the plugin file:
Ohai.plugin(:AwsRegion) do
provides "aws"
depends "ec2"
def collect_region
aws Mash.new
aws[:region] = ec2[:placement_availability_zone].gsub(/[a-z]$/,'')
end
collect_data do
aws collect_region[:region]
end
endAnother note, this all worked with chef-client 12.19.36 but after the chef-client 13 was released new amz linux nodes started failing with issues around ohai and it not collecting attributes on first chef-client run when deployed with autoscale script. When knifing a server with same ami image it all works just fine.
Stromweld
commented
7 years ago also for reference here is the autoscale script used to install chef and register node with AWS OpsWorks for chef-automate:
#!/bin/bash
# required settings
NODE_NAME="$(curl --silent --show-error --retry 3 http://169.254.169.254/latest/meta-data/instance-id)" # this uses the EC2 instance ID as the node name
NODE_AZ="$(curl --silent --show-error --retry 3 http://169.254.169.254/latest/meta-data/placement/availability-zone)"
NODE_REGION=${NODE_AZ::-1} # E.g., remove trailing 'b' from us-west-2b
CHEF_SERVER_REGION="us-west-2" # use one of us-east-1, us-west-1, eu-west-1
CHEF_SERVER_NAME="chef-automate" # The name of your Chef Server
CHEF_SERVER_ENDPOINT="chef-automate-example.us-west-2.opsworks-cm.io" # The FQDN of your Chef Server
# optional
CHEF_ORGANIZATION="default" # AWS OpsWorks for Chef Server always creates the organization "default"
NODE_ENVIRONMENT="base-amz-linux-${APP_ENV}" # E.g. development, staging, onebox ...
CHEF_CLIENT_VERSION="" # latest if empty
# recommended: upload the chef-client cookbook from the chef supermarket https://supermarket.chef.io/cookbooks/chef-client
# Use this to apply sensible default settings for your chef-client config like logrotate and running as a service
# you can add more cookbooks in the run list, based on your needs
RUN_LIST="role[base-amz-linux]" # e.g. "recipe[chef-client],recipe[apache2]"
# ---------------------------
set -e -x
CHEF_CA_PATH="/etc/chef/opsworks-cm-ca-2016-root.pem"
CHEF_DIR="/etc/chef"
CHEF_COMMAND="chef-client"
ASSOCIATION_PATH="${CHEF_DIR}/already_associated_when_this_file_present"
aws_cli() {
aws opsworks-cm --region "${CHEF_SERVER_REGION}" --output text "$@" --server-name "${CHEF_SERVER_NAME}"
}
associate_node() {
if [ ! -f ${ASSOCIATION_PATH} ]; then
client_key="${CHEF_DIR}/client.pem"
mkdir ${CHEF_DIR}
( umask 077; openssl genrsa -out "${client_key}" 2048 )
aws_cli associate-node \
--node-name "${NODE_NAME}" \
--engine-attributes \
"Name=CHEF_ORGANIZATION,Value=${CHEF_ORGANIZATION}" \
"Name=CHEF_NODE_PUBLIC_KEY,Value='$(openssl rsa -in "${client_key}" -pubout)'"
fi
}
write_chef_config() {
config_path="${CHEF_DIR}/client.rb"
if [ ! -f ${config_path} ]; then
echo "Writing Chef config..."
(
echo "chef_server_url 'https://${CHEF_SERVER_ENDPOINT}/organizations/${CHEF_ORGANIZATION}'"
echo "node_name '${NODE_NAME}'"
echo "ssl_ca_file '${CHEF_CA_PATH}'"
) >> ${config_path}
fi
key_path="${CHEF_DIR}/encrypted_data_bag_secret"
if [ ! -f ${key_path} ]; then
echo "****Scrubbed****" \
> ${key_path}
fi
}
install_chef_client() {
command -v ${CHEF_COMMAND} >/dev/null 2>&1 || {
if [ ! $? -eq 0 ]; then
echo "Installing Chef client..."
# see: https://docs.chef.io/install_omnibus.html
curl --silent --show-error --retry 3 --location https://omnitruck.chef.io/install.sh | bash -s -- -v "${CHEF_CLIENT_VERSION}"
echo "Installed"
fi
}
}
install_trusted_certs() {
if [ ! -f ${CHEF_CA_PATH} ]; then
echo "Downloading trusted certs"
curl --silent --show-error --retry 3 --location --output "${CHEF_CA_PATH}" \
"https://opsworks-cm-${NODE_REGION}-prod-default-assets.s3.amazonaws.com/misc/opsworks-cm-ca-2016-root.pem"
fi
}
wait_node_associated() {
if [ ! -f ${ASSOCIATION_PATH} ]; then
echo "Awaiting association..."
aws_cli wait node-associated --node-association-status-token "$1"
echo "Associatied"
touch ${ASSOCIATION_PATH}
fi
}
echo "Requesting association" # Can't be in function because it ends up in the output/${node_association_status_token}
node_association_status_token="$(associate_node)"
install_chef_client
write_chef_config
install_trusted_certs
wait_node_associated "${node_association_status_token}"
${CHEF_COMMAND} -r "${RUN_LIST}" -E "${NODE_ENVIRONMENT}"
I'll dig into this further, but you actually don't need this plugin anymore either. Region is an attribute under ec2 now in Chef 13.
Cookbook version
5.0.3
Chef-client version
13.0.118
Platform Details
chef_version=13.0.118 [0m platform=amazon [0m platform_version=2016.09 [0m ruby=ruby 2.4.
Scenario:
aws ec2 hints in ohai convergence
Steps to Reproduce:
using deployment script from AWS OPSworks for Chef Automate on Elastic Beanstalk instance with wrapper cookbook in run list to install aws cookbook and run ec2 hints.
Expected Result:
converged instance
Actual Result:
Application update failed at 2017-04-19T15:29:23Z with exit status 1 and error: container_command ebInit in .ebextensions/justChef.config failed.