search

matschaffer / knife-solo

DEPRECATED: Please consider using https://knife-zero.github.io/, ansible, or visit https://www.chef.io/ for other ideas
MIT License
787 stars 213 forks source link

Knife-solo and encrypted data bags #323

Open LiamK opened 11 years ago

LiamK commented 11 years ago

Hi -- I've gotten back to working on this, and would appreciate some help. As I mentioned before, it's not unlikely that this issue is due to some misunderstanding on my part. Basically, I'm trying to use encrypted data bags, and Chef is not finding the encrypted data bag secret. It's not clear to me how the secret is copied to the target node. I've tried various ways of specifying the encrypted key. The data_bag_key file does exist and contains the key. It's just not in /etc/chef/ where Chef is looking for it.

Thanks in advance for pointing me in the right direction.

Liam

solo.rb on node:

base = File.expand_path('..', __FILE__)

nodes_path                File.join(base, 'nodes')
role_path                 File.join(base, 'roles')
data_bag_path             File.join(base, 'data_bags')
encrypted_data_bag_secret File.join(base, 'data_bag_key')
environment_path          File.join(base, 'environments')
environment               "_default"

cookbook_path []
cookbook_path << File.join(base, 'cookbooks-1') # /home/liam/Documents/xxx/chef-repo/cookbooks
cookbook_path << File.join(base, 'cookbooks-2') # /home/liam/Documents/xxx/chef-repo/site-cookbooks
cookbook_path << File.join(base, 'cookbooks-3') # /home/liam/.rbenv/versions/1.9.3-p385/lib/ruby/gems/1.9.1/gems/knife-solo-0.4.0/lib/knife-solo/resources/patch_cookbooks

.chef/knife.rb on workstation:

cookbook_path ["cookbooks", "site-cookbooks"]
node_path     "nodes"
role_path     "roles"
data_bag_path "data_bags"
#encrypted_data_bag_secret "/etc/chef/encrypted_data_bag_key"
knife[:secret_file] = ".chef/encrypted_data_bag_key"
knife[:berkshelf_path] = "cookbooks"

knife solo cook piadmin@192.168.1.111 -i ~/.ssh/xxxx
Running Chef on 192.168.1.111...
Checking Chef version...
Enter the password for piadmin@192.168.1.111: 
Installing Berkshelf cookbooks to 'cookbooks'...
Using application (2.0.4)
Using application_nginx (1.0.4)
Using build-essential (1.4.0)
Using chef-solo-search (0.5.1)
Using database (1.5.2)
Using ntp (1.3.2)
Using postgresql (3.1.0)
Using users (1.6.0)
Installing repmgr (0.2.2) from git: 'git@github.com:LiamK/repmgr.git' with branch: 'c6961b561a99f95502c9cc0fb34a6e4cc7ea8010'
Installing nginx (1.7.1) from git: 'git@github.com:LiamK/nginx.git' with branch: '00fa1ab5c6a7ed3a89403e5c98c4798005bc9124'
Installing application_python (1.2.5) from git: 'git@github.com:LiamK/application_python.git' with branch: 'd10fbeba71fc1b056cd3be6ecb499566c50b1cc7'
Using mysql (3.0.0)
Using openssl (1.0.2)
Using aws (0.101.0)
Using xfs (1.1.0)
Using apt (1.10.0)
Using discovery (0.2.0)
Using runit (1.1.4)
Using yum (2.2.2)
Using ohai (1.1.8)
Using python (1.4.0)
Using gunicorn (1.1.0)
Using supervisor (0.4.2)
Uploading the kitchen...
Generating solo config...
Running Chef...

Starting Chef Client, version 11.4.4
Compiling Cookbooks...
[2013-11-27T13:37:13-08:00] WARN: Chef::Mixin::RecipeDefinitionDSLCore is deprecated. Use Chef::DSL::Recipe instead.

[2013-11-27T13:37:13-08:00] WARN: Called from: 
    /home/piadmin/chef-solo/cookbooks-1/application/resources/default.rb:23:in `class_from_file'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_eval'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_from_file'
[2013-11-27T13:37:13-08:00] WARN: Chef::Mixin::LanguageIncludeRecipe is deprecated, use Chef::DSL::Recipe
instead.

[2013-11-27T13:37:13-08:00] WARN: Called from: 
    /home/piadmin/chef-solo/cookbooks-1/application_nginx/providers/nginx_load_balancer.rb:21:in `class_from_file'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_eval'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_from_file'
[2013-11-27T13:37:14-08:00] WARN: Chef::Mixin::LanguageIncludeRecipe is deprecated, use Chef::DSL::Recipe
instead.

[2013-11-27T13:37:14-08:00] WARN: Called from: 
    /home/piadmin/chef-solo/cookbooks-1/application_python/providers/celery.rb:21:in `class_from_file'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_eval'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_from_file'
[2013-11-27T13:37:14-08:00] WARN: Chef::Mixin::LanguageIncludeRecipe is deprecated, use Chef::DSL::Recipe
instead.

[2013-11-27T13:37:14-08:00] WARN: Called from: 
    /home/piadmin/chef-solo/cookbooks-1/application_python/providers/django.rb:23:in `class_from_file'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_eval'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_from_file'
[2013-11-27T13:37:14-08:00] WARN: Chef::Mixin::LanguageIncludeRecipe is deprecated, use Chef::DSL::Recipe
instead.

[2013-11-27T13:37:14-08:00] WARN: Called from: 
    /home/piadmin/chef-solo/cookbooks-1/application_python/providers/gunicorn.rb:23:in `class_from_file'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_eval'
    /usr/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/mixin/from_file.rb:42:in `class_from_file'
Recipe: build-essential::debian
  * execute[apt-get-update-build-essentials] action run (skipped due to not_if)
  * package[autoconf] action install (up to date)
  * package[binutils-doc] action install (up to date)
  * package[bison] action install (up to date)
  * package[build-essential] action install (up to date)
  * package[flex] action install (up to date)
[2013-11-27T13:37:14-08:00] WARN: Cloning resource attributes for package[libpq-dev] from prior resource (CHEF-3694)
[2013-11-27T13:37:14-08:00] WARN: Previous package[libpq-dev]: /home/piadmin/chef-solo/cookbooks-1/postgresql/recipes/client.rb:36:in `block in from_file'
[2013-11-27T13:37:14-08:00] WARN: Current  package[libpq-dev]: /home/piadmin/chef-solo/cookbooks-1/repmgr/recipes/source_install.rb:8:in `from_file'
Recipe: repmgr::source_install
  * package[libpq-dev] action install (up to date)
[2013-11-27T13:37:14-08:00] WARN: Cloning resource attributes for package[rsync] from prior resource (CHEF-3694)
[2013-11-27T13:37:14-08:00] WARN: Previous package[rsync]: /home/piadmin/chef-solo/cookbooks-1/repmgr/recipes/default.rb:3:in `from_file'
[2013-11-27T13:37:14-08:00] WARN: Current  package[rsync]: /home/piadmin/chef-solo/cookbooks-1/repmgr/recipes/source_install.rb:16:in `block in from_file'

================================================================================
Recipe Compile Error in /home/piadmin/chef-solo/cookbooks-1/repmgr/recipes/default.rb
================================================================================

Errno::ENOENT
-------------
No such file or directory - file not found '/etc/chef/encrypted_data_bag_secret'

Cookbook Trace:
---------------
  /home/piadmin/chef-solo/cookbooks-1/repmgr/recipes/configure.rb:58:in `from_file'
  /home/piadmin/chef-solo/cookbooks-1/repmgr/recipes/default.rb:7:in `from_file'

Relevant File Content:
----------------------
/home/piadmin/chef-solo/cookbooks-1/repmgr/recipes/configure.rb:

 51:    )
 52:    mode 0600
 53:  end
 54:  
 55:  
 56:  if(node[:repmgr][:data_bag][:encrypted])
 57:    if(node[:repmgr][:data_bag][:secret])
 58>>     secret = Chef::EncryptedDataBagItem.load_secret(node[:repmgr][:data_bag][:secret])
 59:    end
 60:    key_bag = Chef::EncryptedDataBagItem.load(
 61:      node[:repmgr][:data_bag][:name],
 62:      node[:repmgr][:data_bag][:item],
 63:      secret
 64:    )
 65:  else
 66:    key_bag = data_bag_item(node[:repmgr][:data_bag][:name], node[:repmgr][:data_bag][:item])
 67:  end

[2013-11-27T13:37:14-08:00] ERROR: Running exception handlers
[2013-11-27T13:37:14-08:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated
[2013-11-27T13:37:14-08:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-11-27T13:37:14-08:00] FATAL: Errno::ENOENT: No such file or directory - file not found '/etc/chef/encrypted_data_bag_secret'
ERROR: RuntimeError: chef-solo failed. See output above.
LiamK commented 11 years ago

Okay, I've made some progress. Sometimes you just need to take a fresh look at things. I added in the node[:repmgr][:data_bag][:secret] which is a path to a file on the node containing the key. Now it's working. I had to specify the full path. /home/piadmin/chef-solo/data_bag_key Is there a better way to handle this?
Sometimes the path might be different if installing via some other user -- the piadmin part would be root or something else.

tmatilai commented 10 years ago

@LiamK I guess the most common case is that you have the key in your workstation (but not stored to git repo ;)), and specify encrypted_data_bag_secret in .chef/knife.rb. Then knife-solo uploads the key and sets encrypted_data_bag_secret in the solo.rb. You have now the option commented out in your knife.rb.