search

chef-cookbooks / supermarket-omnibus-cookbook

A cookbook for installing supermarket from Omnibus (deb, rpm) packages
Apache License 2.0
17 stars 29 forks source link

Only converges successfully when chef-client run by hand + HOME env variable #41

Closed jorhett closed 8 years ago

jorhett commented 8 years ago

If a machine is booted up and runs chef-client with this cookbook it will not converge, as shown below. If you log in and run it by hand, it will converge that time, and then fail to converge when it runs as a daemon again. It apparently needs a HOME environment variable.

[2016-07-01T17:35:30+00:00] INFO: Processing link[/opt/supermarket/embedded/service/supermarket/config/database.yml] action create (omnibus-supermarket::database line 70)
[2016-07-01T17:35:30+00:00] INFO: Processing file[/opt/supermarket/embedded/service/supermarket/db/schema.rb] action create (omnibus-supermarket::database line 76)
[2016-07-01T17:35:30+00:00] INFO: Processing execute[database schema] action run (omnibus-supermarket::database line 80)
ESC[0m
================================================================================ESC[0m
ESC[31mError executing action `run` on resource 'execute[database schema]'ESC[0m
================================================================================ESC[0m

ESC[0mMixlib::ShellOut::ShellCommandFailedESC[0m
------------------------------------ESC[0m
Expected process to exit with [0], but received '1'
ESC[0m---- Begin output of bundle exec rake db:migrate db:seed ----
ESC[0mSTDOUT:
ESC[0mSTDERR: /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rbreadline.rb:1097:in `<module:RbReadline>': HOME environment variable (or HOMEDRIVE and HOMEPATH) must be set and point to a directory (RuntimeError)
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rbreadline.rb:17:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/readline.rb:10:in `<module:Readline>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/readline.rb:8:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.12.5/lib/bundler/vendor/thor/lib/thor/line_editor/readline.rb:2:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.12.5/lib/bundler/vendor/thor/lib/thor/line_editor.rb:2:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.12.5/lib/bundler/vendor/thor/lib/thor/base.rb:8:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.12.5/lib/bundler/vendor/thor/lib/thor.rb:2:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.12.5/lib/bundler/vendored_thor.rb:3:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.12.5/lib/bundler/friendly_errors.rb:4:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
ESC[0m  from /opt/supermarket/embedded/lib/ruby/gems/2.1.0/gems/bundler-1.12.5/exe/bundle:18:in `<top (required)>'
ESC[0m  from /opt/supermarket/embedded/bin/bundle:23:in `load'
ESC[0m  from /opt/supermarket/embedded/bin/bundle:23:in `<main>'
ESC[0m---- End output of bundle exec rake db:migrate db:seed ----
ESC[0mRan bundle exec rake db:migrate db:seed returned 1ESC[0m

ESC[0mResource Declaration:ESC[0m
---------------------ESC[0m
# In /var/opt/supermarket/cache/cache/cookbooks/omnibus-supermarket/recipes/database.rb
ESC[0m
ESC[0m 80: execute 'database schema' do
ESC[0m 81:   command 'bundle exec rake db:migrate db:seed'
ESC[0m 82:   cwd node['supermarket']['app_directory']
ESC[0m 83:   env 'RAILS_ENV' => 'production'
ESC[0m 84:   user node['supermarket']['user']
ESC[0m 85: end
ESC[0m
ESC[0mCompiled Resource:ESC[0m
------------------ESC[0m
# Declared in /var/opt/supermarket/cache/cache/cookbooks/omnibus-supermarket/recipes/database.rb:80:in `from_file'
ESC[0m
ESC[0mexecute("database schema") do
ESC[0m  action "run"
ESC[0m  retries 0
ESC[0m  retry_delay 2
ESC[0m  default_guard_interpreter :execute
ESC[0m  command "bundle exec rake db:migrate db:seed"
ESC[0m  backup 5
ESC[0m  cwd "/opt/supermarket/embedded/service/supermarket"
ESC[0m  environment {"RAILS_ENV"=>"production"}
ESC[0m  returns 0
ESC[0m  user "supermarket"
ESC[0m  declared_type :execute
ESC[0m  cookbook_name "omnibus-supermarket"
ESC[0m  recipe_name "database"
ESC[0mend
ESC[0m
ESC[0m[2016-07-01T17:35:30+00:00] INFO: Running queued delayed notifications before re-raising exception
robbkidd commented 8 years ago

@jorhett What version of the supermarket package was being installed? On what platform? I'm trying to replicate.

robbkidd commented 8 years ago

So far, I've been unable to replicate the error with the test-kitchen config for this cookbook. But I still suspect there is a problem given that I had to set HOME in the service startup scripts after upgrading some components in the omnibus install.

irvingpop commented 8 years ago

Initial analysis is that it isn't a bug in this cookbook, but points to the omnibus-supermarket::database recipe embedded in the omnibus here and possibly deeper to that rake task.

robbkidd commented 8 years ago

Yes, true. The problem probably lay in the cookbooks internal to the omnibus install. I've been trying to replicate the problem from up on high in this cookbook however.

jorhett commented 8 years ago

supermarket 2.7.4-1 on Ubuntu 14.04. I can send you the entire log in chef-success slack if you like...

nellshamrell commented 8 years ago

Looking into this now.

nellshamrell commented 8 years ago

Steps I've currently take to replicate this (this is based on how we currently spin up supermarket nodes in Chef): 1) Spun up new Ubuntu 14.04 node in AWS 2) Bootstrap new node with knife bootstrap

$ (my-workstation) knife node bootstrap node_public_ip -i my_ssh_key_path -N supermarket-node2 -x ubuntu --sudo

3) Add the default recipe of our Supermarket wrapper cookbook to the node run list

$ (my-workstation) knife node run list add supermarket-node2 'recipe[supermarket-wrapper::default]'

4) SSH'd into the new node

$  (my-workstation) ssh -i my_aws_key ubuntu@node_ip

5) Ran chef client on the node

$  (supermarket-node2) sudo chef-client

It configured successfully and installed supermarket 2.7.4 6) Tried running chef-client again

$  (supermarket-node2) sudo chef-client

7) Ran successfully

@jorhett could you detail the steps you took to install Supermarket? And I would love to see the log, you can send it to me on Chef Success Slack @nshamrell

nellshamrell commented 8 years ago

Hi @jorhett! Closing this for now because we haven't heard back from you. If this is still a problem, feel free to re-open this with some more detailed steps about when you see this error, this will help us replicate it. Thanks!

jorhett commented 8 years ago

Sorry, I provided all the information in the chef-success slack channel as mentioned above, as some internal corporate details were in the data. Could you tell me what else you were looking for?

jorhett commented 8 years ago

@nellshamrell And your test case was invalid, for the reasons I specified in the first sentence of the problem report. If you run Chef by hand then the process is provided with a HOME environment variable and succeeds. This error occurs when the node boots and runs Chef as a daemon without a human being involved (you know, automation...)

irvingpop commented 8 years ago

@jorhett Please remember that you're commenting on a community-supported Open Source project, and snarky comments (or those interpreted as such) don't elicit generosity in those who are in a position to help you. Please take a look at the Chef Community Guidelines as a reference.

Despite that, we want to make sure you're successful. I dug into the issue and here's what I found:

The issue you're hitting is one that I've personally dealt with, as will likely most people who automate. Running CI agent that don't set a HOME for jobs will cause you endless problems, and you should consider:

As this isn't a problem with this cookbook, nor the Supermarket omnibus package, I have to agree with Nell that this isn't the right place to report the issue (and is also an invalid issue because the issue goes way below any software that Chef maintains)

jorhett commented 8 years ago

I'm honestly sorry. That wasn't intended as snark, although (in retrospect) I see why you thought it that way. It was more tired self-amusement if anything, meant in a "you know" way.

I disagree with your assessment, however, in that I am using a Chef-supported and maintained cookbook and process which is intended to be a black box for us. I am using the cookbook and setting attributes as documented. The only way to fix this problem is to abandon your provided cookbook, or to retroactively modify resources in your cookbook. Is that not a bug?

The bug is that your supported cookbook is failing to set the HOME environment when calling that process. My bug report is that you are failing to use your own advice above. Again, I've no attempt or desire to be snarky here. I'm being entirely serious and straightforward.