Question: Error when using GD to upload with Berkshelf

brianwallace commented 8 years ago

I am attempting to setup GD to upload cookbooks from a chef-repo to self-hosted chef server. We use bershelf to manage cookbook dependencies so I also have berks set to true in the GD config.

But I continue to hit an error on the berks upload that I can't work through. I am wondering if anyone has any pointers?

The error I am hitting is as follows. When I run the command printed at the bottom in shell everything succeeds.

> bundler exec grocery-delivery -c gd-config.rb -v
WARN: Attempting to acquire lock
WARN: Lock acquired
WARN: Updating repo
INFO: Running "git pull --rebase"
WARN: Uploading all cookbooks with berkshelf
WARN: Running berkshelf on cookbook: /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook
INFO: Running: cd /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook && /opt/chefdk/bin/berks install --config=/Users/bwallace/github/grocery-delivery/config.json && /opt/chefdk/bin/berks upload --config=/Users/bwallace/github/grocery-delivery/config.json
INFO: STDERR: /Users/bwallace/.rvm/gems/ruby-2.2.1/gems/bundler-1.11.2/lib/bundler/rubygems_integration.rb:304:in `block in replace_gem': addressable is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
INFO: STDERR: from /opt/chefdk/bin/berks:4:in `<main>'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/gems/mixlib-shellout-2.2.6/lib/mixlib/shellout.rb:289:in `invalid!'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/gems/mixlib-shellout-2.2.6/lib/mixlib/shellout.rb:276:in `error!'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/util.rb:42:in `exec!'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:88:in `block (2 levels) in berks_cookbook_upload_all'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:86:in `each'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:86:in `block in berks_cookbook_upload_all'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:84:in `each'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:84:in `berks_cookbook_upload_all'
ERROR: /Users/bwallace/github/grocery-delivery/bin/grocery-delivery:74:in `full_upload'
ERROR: /Users/bwallace/github/grocery-delivery/bin/grocery-delivery:166:in `upload_changed'
ERROR: /Users/bwallace/github/grocery-delivery/bin/grocery-delivery:277:in `<top (required)>'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bin/grocery-delivery:23:in `load'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bin/grocery-delivery:23:in `<main>'
WARN: Expected process to exit with [0], but received '1'
---- Begin output of cd /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook && /opt/chefdk/bin/berks install --config=/Users/bwallace/github/grocery-delivery/config.json && /opt/chefdk/bin/berks upload --config=/Users/bwallace/github/grocery-delivery/config.json ----

I should note that I am uploading to a chef-server with a self-signed cert. I have turned ssl-validation off by using the [undocumented] berks_config parameter to give berkshelf the following custom config:

{
  "ssl": {
    "verify": false
  }
}

jaymzh commented 8 years ago

It's a bit hard for us to debug this since we don't use berkshelf - someone else added the berks code. That said - you seemed to cut the error message off before the actual output... what's the output it got from the berks command?

brianwallace commented 8 years ago

Hi @jaymzh, thanks for taking a look even though you don't use berks or wrote the code.

The full berks output is as follows -- nothing else that seems too enlightening to me.

---- Begin output of cd /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook && /opt/chefdk/bin/berks install --config=/Users/bwallace/github/grocery-delivery/config.json && /opt/chefdk/bin/berks upload --config=/Users/bwallace/github/grocery-delivery/config.json ----
STDOUT:
STDERR: /Users/bwallace/.rvm/gems/ruby-2.2.1/gems/bundler-1.11.2/lib/bundler/rubygems_integration.rb:304:in `block in replace_gem': addressable is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
    from /opt/chefdk/bin/berks:4:in `<main>'
---- End output of cd /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook && /opt/chefdk/bin/berks install --config=/Users/bwallace/github/grocery-delivery/config.json && /opt/chefdk/bin/berks upload --config=/Users/bwallace/github/grocery-delivery/config.json ----

jaymzh commented 8 years ago

OK, so it looks like berks itself doesn't work inside your bundle. I don't know how you created your bundle, but GD is just running berks which depends on addressable which isn't there.

bundle install addressable may fix you up, but the tl;dr is bundle exec berks needs to work for this to work.

brianwallace commented 8 years ago

@jaymzh, thanks for the tips. I think you pointed me on the right track and that there were missing dependencies in my gemset.

I eventually was able to get bundle exec berks to run successfully by adding gem 'berkshelf', '~> 4.2.0' into the GD Gemfile. It seemed that this was the easiest way to ensure that all deps were met. This was in a clean rvm environment with a fresh gemset.

But I still ran into further berks dependency issues. I think the berks gem in my "grocery_delivery" gemset is conflicting with the chefdk berks version I have installed.

---- Begin output of cd /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook && /opt/chefdk/bin/berks install --config=/Users/bwallace/github/grocery-delivery/config.json && /opt/chefdk/bin/berks upload --config=/Users/bwallace/github/grocery-delivery/config.json ----
STDOUT:
STDERR: /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/dependency.rb:319:in `to_specs': Could not find 'berkshelf' (= 4.2.0) - did find: [berkshelf-4.2.3] (Gem::LoadError)
Checked in 'GEM_PATH=/Users/bwallace/.rvm/gems/ruby-2.2.0@grocery_delivery:/Users/bwallace/.rvm/gems/ruby-2.2.0@global', execute `gem env` for more information
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/dependency.rb:328:in `to_spec'
    from /opt/chefdk/embedded/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb:1028:in `find_by_name'
    from /opt/chefdk/bin/berks:45:in `<main>'
---- End output

And ChefDK info:

> chef --version
Chef Development Kit Version: 0.11.2
chef-client version: ERROR
berks version: ERROR
kitchen version: 1.5.0

At this point in time I am going to close this issue/question as I think the root cause are all related to dependencies within my environment. I having a feeling that there may be a GD gem dependency that is missing form the Gemfile, but until I get this working I won't know for sure.

I'll update this issue if I get it working.

brianwallace commented 8 years ago

@wjimenez5271 Looks like you may have the been the original contributor of the GD berkshelf code. If you have any tips on what may be going wrong with my berkshelf dependencies, I would definitely appreciate it!

wjimenez5271 commented 8 years ago

@brianwallace the berkshelf stuff doesn't do much other than call the berkshelf resolver to do its thing. It is calling the berkshelf CLI binary, so GD doesn't need to be aware of any berkshelf ruby libraries or gems, only the path the the bin.

I would double check that your pointing to the right berkshelf binary (if you are using ChefDK that will likely be somewhere in /opt/chefdk) in the Grocery Delivery config:

https://github.com/facebook/grocery-delivery/blob/master/bin/grocery-delivery#L153

facebook / grocery-delivery

Question: Error when using GD to upload with Berkshelf #27