Dependency Hell - Githubissues

pikesley commented 11 years ago

Everything 'splodes. Got the latest 2.1.0.rc.5, built and configured everything, then did this:

[sam@dirac:~/ODI/cuke-chef] cucumber-chef setup
cucumber-chef v2.1.0.rc.5
Creating VAGRANT instance completed in 26.4955 seconds.
Bootstrapping VAGRANT instance completed in 699.5953 seconds.
Waiting for the chef-server completed in 0.1017 seconds.
Waiting for the chef-server-webui completed in 0.1020 seconds.
Downloading chef-server credentials completed in 0.5072 seconds.
Building 'cc-knife' configuration completed in 0.1022 seconds.
Uploading 'cucumber-chef' cookbooks completed in 3.2764 seconds.
Uploading 'cucumber-chef' roles completed in 1.0106 seconds.
Tagging 'cucumber-chef' node completed in 1.2221 seconds.
Setting 'cucumber-chef' run list completed in 1.1112 seconds.
Performing chef-client run /Users/sam/.rvm/gems/ruby-1.9.2-p320@cuke-chef/gems/ztk-0.3.1/lib/ztk/base.rb:90:in `log_and_raise': exit_code=1 (ZTK::SSHError)
    from /Users/sam/.rvm/gems/ruby-1.9.2-p320@cuke-chef/gems/ztk-0.3.1/lib/ztk/base.rb:121:in `log_and_raise'

and in the logs, I got:

STDERR: ERROR:  While executing gem ... (Gem::DependencyError)
    Unable to resolve dependencies: cucumber requires json (>= 1.4.6); gherkin requires json (>= 1.7.6); chef requires json (<= 1.6.1, >= 1.4.4); vagrant requires json (~> 1.5.1)

Does this work for anybody? Am I doing something dumb? Going to try rc4, see if that works any better.

Thanks

Sam

zpatten commented 11 years ago

This seems to be a Chef dep issue; someone else hit this as well as me. I haven't verified but it was claimed in another issue that Chef had too tight of dep on JSON in one of the recent releases.

https://github.com/Atalanta/cucumber-chef/issues/93

pikesley commented 11 years ago

I have chef 10.18.2 as referenced in #93, still failing. I don't think it's that.

zpatten commented 11 years ago

Hmmm; can you send me your full logs from that run? Or gist them and link them here? If you have too much private info in them you can just email them to me directly; zachary at jovelabs dot com.

zpatten commented 11 years ago

I'm going to try and reproduce in the mean time.

zpatten commented 11 years ago

Also send you displayconfig output (be sure to rip out any AWS or other sensitive data).

pikesley commented 11 years ago

running it again now, will Gist it when it dies

zpatten commented 11 years ago

Awesome; thank you sir!

zpatten commented 11 years ago

Ya the worse part of this whole JSON dep issue is that it is not even a dependency for Cucumber-Chef; so it's other deps fighting over it the JSON version... :(

zpatten commented 11 years ago

You might also ssh into the test lab and grab the chef-client log; should be in /var/log/chef/client.log. There is a "bootstrap" option to the ssh command to go in as the bootstrap user/creds.

$ bin/cucumber-chef help ssh
Usage:
  cucumber-chef ssh [container]

Options:
  [--bootstrap]  # Use the bootstrap settings.

SSH to cucumber-chef test lab or [container] if specified.

zpatten commented 11 years ago

Ya something is going wonky; I'm hitting this now too all of the sudden.

Unable to resolve dependencies: vagrant requires json (~> 1.5.1); cucumber requires json (>= 1.4.6); chef requires json (<= 1.6.1, >= 1.4.4); gherkin requires json (>= 1.7.6)

Seems Vagrant, Chef and Gherkin are in a tug of war...

zpatten commented 11 years ago

Not to mention this part of it all is very bad: chef requires json (<= 1.6.1, >= 1.4.4); gherkin requires json (>= 1.7.6) Looks like I need to figure out what gem broken everything and pin it back.

pikesley commented 11 years ago

Ah, OK, at least it's not just me :)

I did see something on the Twitters about rubygems.org having been compromised earlier. Could that be kicking us here?

Still want these huge gists? Looks like you're hitting the same pain as me now.

pikesley commented 11 years ago

OK, gists are here: https://gist.github.com/844ecb5bfa064f5816b4

I gotta run now, I'll be back in the morning (London time)

Thanks :)

zpatten commented 11 years ago

Awesome; thank your sir. I'm going to do some digging into this today. I'm starting to wonder if this has anything to do with the rubygems compromise. I successfully built labs everyday this week until today. And now this weird JSON dep issue again.

Either way I will hopefully have something concrete to report by EOD.

zpatten commented 11 years ago

Gherkin went from 2.11.5 -> 2.11.6 yesterday. With this

https://rubygems.org/gems/gherkin/versions/2.11.5 Runtime Dependencies is json >= 1.4.6

https://rubygems.org/gems/gherkin/versions/2.11.6 Runtime Dependencies is json >= 1.7.6

I've added Gherkin to the CC gemfile and pined it at <= 2.11.5. This seems to have resolved the issue for me.

Once I have the new RC out and build a test lab I'll report my findings here; but all local testing confirms it's fixed.

zpatten commented 11 years ago

OK; let me revise; I'll push the new RC out as soon as RubyGems re-enabled pushing.

pikesley commented 11 years ago

So I have RC6. The catastrophic fail of yesterday has gone (thanks for that :) ), but I have some New And Interesting fail now:

[sam@dirac:~/ODI/cuke-chef] cucumber-chef setup
cucumber-chef v2.1.0.rc.6
Creating VAGRANT instance completed in 21.5436 seconds.
Bootstrapping VAGRANT instance completed in 799.3118 seconds.
Waiting for the chef-server completed in 0.1023 seconds.
Waiting for the chef-server-webui completed in 0.1011 seconds.
Downloading chef-server credentials completed in 0.6083 seconds.
Building 'cc-knife' configuration completed in 0.1023 seconds.
Uploading 'cucumber-chef' cookbooks completed in 2.0356 seconds.
Uploading 'cucumber-chef' roles completed in 0.6074 seconds.
Tagging 'cucumber-chef' node completed in 0.7072 seconds.
Setting 'cucumber-chef' run list completed in 0.7054 seconds.
Performing chef-client run /Users/sam/.rvm/gems/ruby-1.9.3-p362@cuke-chef/gems/ztk-0.3.1/lib/ztk/base.rb:90:in `log_and_raise': exit_code=1 (ZTK::SSHError)
    from /Users/sam/.rvm/gems/ruby-1.9.3-p362@cuke-chef/gems/ztk-0.3.1/lib/ztk/base.rb:121:in `log_and_raise'
    from /Users/sam/.rvm/gems/ruby-1.9.3-p362@cuke-chef/gems/ztk-0.3.1/lib/ztk/ssh.rb:315:in `exec'
<snip>

and in the (local) cucumber-chef log:

[Thu, 31 Jan 2013 15:54:33 +0000] DEBUG: gem_package[cucumber-chef] using gem '/usr/bin/gem'
[Thu, 31 Jan 2013 15:54:33 +0000] DEBUG: gem_package[cucumber-chef] no installed version found for cucumber-chef (= 2.1.0.rc.6)

================================================================================
Error executing action `install` on resource 'gem_package[cucumber-chef]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '2'
---- Begin output of /usr/bin/gem install cucumber-chef -q --no-rdoc --no-ri -v "2.1.0.rc.6" --prerelease ----
STDOUT:
STDERR: ERROR:  Could not find a valid gem 'cucumber-chef' (= 2.1.0.rc.6) in any repository
ERROR:  Possible alternatives: cucumber-chef
---- End output of /usr/bin/gem install cucumber-chef -q --no-rdoc --no-ri -v "2.1.0.rc.6" --prerelease ----
Ran /usr/bin/gem install cucumber-chef -q --no-rdoc --no-ri -v "2.1.0.rc.6" --prerelease returned 2

Resource Declaration:
---------------------
# In /var/cache/chef/cookbooks/cucumber-chef/recipes/test_lab.rb

122: gem_package "cucumber-chef" do
123:   gem_binary("/usr/bin/gem")
124:
125:   version(node['cucumber_chef']['version'])
126:
127:   if node['cucumber_chef']['prerelease']
128:     options("--prerelease")
129:   end
130: end

Compiled Resource:
------------------
# Declared in /var/cache/chef/cookbooks/cucumber-chef/recipes/test_lab.rb:122:in `from_file'

gem_package("cucumber-chef") do
  provider Chef::Provider::Package::Rubygems
  version "2.1.0.rc.6"
  recipe_name "test_lab"
  package_name "cucumber-chef"
  action :install
  retry_delay 2
  gem_binary "/usr/bin/gem"
  cookbook_name "cucumber-chef"
  options "--prerelease"
  retries 0
end

[Thu, 31 Jan 2013 15:54:41 +0000] DEBUG: Re-raising exception: Mixlib::ShellOut::ShellCommandFailed - gem_package[cucumber-chef] (cucumber-chef::test_lab line 122) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
---- Begin output of /usr/bin/gem install cucumber-chef -q --no-rdoc --no-ri -v "2.1.0.rc.6" --prerelease ----
STDOUT:
STDERR: ERROR:  Could not find a valid gem 'cucumber-chef' (= 2.1.0.rc.6) in any repository
ERROR:  Possible alternatives: cucumber-chef
---- End output of /usr/bin/gem install cucumber-chef -q --no-rdoc --no-ri -v "2.1.0.rc.6" --prerelease ----

There's more if you want it. It's like it's trying to install cucumber-chef 2.1.0.rc.6 from Rubygems, which plainly isn't going to work.

I'm inclined to think this one is me being dumb, but I can't see what I've done (or not done). Any clues?

zpatten commented 11 years ago

Ya the test lab is going to try and pull RC6 from RubyGems; so until they turn their Push API back on we're dead in the water.

I will say this; you can try using cc-push (command line binary that comes with cucumber-chef). That will build the gem locally and push it up into your test lab (I used this command for testing changes without push new RC's out). If you can get cc-push to run fine; you should be able to re-run the setup (do not destroy the lab thou!); and it will continue.

I made setup idempotent; you can run it as many times as you want against even against an existing live test lab. It should do nothing destructive if you have a working setup already.

pikesley commented 11 years ago

If you ever come to London, @zpatten, I'm going to buy you so much beer :)

So this now works. BUT when I try to run some actual cukes:

[sam@dirac:~/ODI/cuke-chef] cucumber
Unable to activate vagrant-1.0.6, because json-1.6.1 conflicts with json (~> 1.5.1) (Gem::LoadError)

zpatten commented 11 years ago

@pikesley :) I'll take you up on that for sure :)

hmmmm; man I'm about to get medieval on some gem deps UGH!

rant I don't understand why so many of these gems put such tight restrictions on core gems like JSON. As if to assume their gem is the only one you'll be using; god forbid you might use another gem with it!

It's called: write a JSON wrapper class people; put all your ugly JSON::VERSION switch type logic in there, get rid of your version constraints that keep me stuck in the past! How do you think people write cross-platform code FFS! JSON 1.5.1 to 1.61 isn't even a major version change! facepalm.

Not being backward compatible: OK
Not being future capable: JUST PLAIN STUPID /rant

I'll try to reproduce that on my end. My gut says you might have to add JSON to your chef-repo Gemfile and put a version constraint on it to keep it from going too high.

pikesley commented 11 years ago

If adding JSON to my chef-repo Gemfile will make this work, that's what I'll do.

Except my chef-repo has no Gemfile. Am I missing something here?

YungSang commented 11 years ago

It seems that cc-push can be used only after cucumber-chef setup is completed properly. So at this time, while "RubyGems.org is in read-only mode.", cf) http://rubygems.org/ We have to install the old revision first, then can upgrade to rc.6 using cc-push.

To setup rc.5 with vagrant. I wrote an instruction under this situation. https://gist.github.com/4678691

zpatten commented 11 years ago

Awesome; thanks! Ya; slipped my mind the test lab key generation happens in the chef-run; so @YungSang is quite correct. Thanks again for putting the gist together. I will see about re-factoring cc-push to use the bootstrap credentials so it can be used anytime.

@pikesley Ya I setup all my chef-repo's with bundler & Gemfiles. There's nothing wrong with using the system gems and skipping bundler; but I have to switch around through so many repos all the time I keep it all separate with RVM & bundler so I don't have to worry about gem bleed between projects. Also using a Gemfile will afford the ability to do the tricks as YungSang illustrates in that Gist. There is also a few gem development patterns you can apply to a Gemfile that ease the development in situations like this.

YungSang commented 11 years ago

@zpatten my pleasure. I'm glad it is helpful for you.

zpatten commented 11 years ago

rc6 is released now

pikesley commented 11 years ago

This is great :) I can now build me a test lab:

Your test lab has now been provisioned!  Enjoy!

+-------------------------------------------------------------------+
|                      PROVIDER: Cucumber::Chef::Provider::Vagrant  |
|                            ID: default                            |
|                         STATE: running                            |
|                      USERNAME: vagrant                            |
|                    IP ADDRESS: 127.0.0.1                          |
|                          PORT: 2222                               |
|               CHEF-SERVER API: http://127.0.0.1:4000              |
|             CHEF-SERVER WEBUI: http://127.0.0.1:4040              |
|      CHEF-SERVER DEFAULT USER: admin                              |
|  CHEF-SERVER DEFAULT PASSWORD: p@ssw0rd1                          |
+-------------------------------------------------------------------+

However I still get the Unable to activate vagrant-1.0.6, because json-1.6.1 conflicts with json (~> 1.5.1) (Gem::LoadError) thing when I try to run some actual cukes.

Can somebody tell me how to employ a Gemfile and bundler to get around this please (if I've even understood that correctly)? I'm kind of in over my head here ;)

YungSang commented 11 years ago

I don't see any such error messages here on my Mac.

I use $ bundle exec cucumber instead of $ cucumber at my project home directory.

I will gist my procedure.

pikesley commented 11 years ago

Hey dude, thanks for helping with this.

So I run it with bundle exec cucumber and the first time, I get this:

[sam@dirac:~/ODI/cuke-chef] bundle exec cucumber
/Users/sam/.rvm/gems/ruby-1.9.3-p362@global/gems/bundler-1.2.3/lib/bundler/rubygems_integration.rb:147:in `block in replace_gem': cucumber is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
    from /Users/sam/.rvm/gems/ruby-1.9.3-p362@cuke-chef/bin/cucumber:18:in `<main>'
    from /Users/sam/.rvm/gems/ruby-1.9.3-p362@cuke-chef/bin/ruby_noexec_wrapper:14:in `eval'
    from /Users/sam/.rvm/gems/ruby-1.9.3-p362@cuke-chef/bin/ruby_noexec_wrapper:14:in `<main>'

So that's OK, I add "cucumber" to my Gemfile, and now:

[sam@dirac:~/ODI/cuke-chef] bundle exec cucumber
cannot load such file -- rspec/expectations (LoadError)
/Users/sam/ODI/cuke-chef/features/support/env.rb:1:in `require'
/Users/sam/ODI/cuke-chef/features/support/env.rb:1:in `<top (required)>'
<snip>

rspec? WTF?

I can't help feeling I'm getting really close to having this working now...

YungSang commented 11 years ago

You don't need cucumber and rspec in the Gemfile, but cucumber-chef only to use cucumber-chef and cucumber.

Gemfile: source "https://rubygems.org" gem "cucumber-chef", :git => "git://github.com/Atalanta/cucumber-chef", :tag => "v2.1.0.rc.6"

then to install cucumber-chef into an isolated env. $ bundle --path=vendor/bundle

https://gist.github.com/4703835

pikesley commented 11 years ago

Oh wow it works! Well, kind of. It complained about

[sam@dirac:~/ODI/cuke-chef] bundle exec cucumber
  * cucumber-chef v2.1.0.rc.6
uninitialized constant Class::Librarian (NameError)
/Users/sam/ODI/cuke-chef/.cucumber-chef/vagrant/knife.rb:16:in `from_file'
/Users/sam/.rvm/gems/ruby-1.9.3-p362@cuke-chef/gems/mixlib-config-1.1.2/lib/mixlib/config.rb:37:in `instance_eval'

so I edited .cucumber-chef/vagrant/knife.rb and commented out these two lines:

#require 'librarian/chef/integration/knife'
#cookbook_path Librarian::Chef.install_path, "#{current_dir}/../site-cookbooks"

and now it actually fires up a test instance and attempts to run tests! And then it dies with

    * the following cookbooks have been uploaded:                            # cucumber-chef-92bb36e5ad09/lib/cucumber/chef/steps/chef_steps.rb:57
      | cookbook    | cookbook_path |
      | chef-client | ./cookbooks   |
      exec("/usr/bin/env knife cookbook upload chef-client -o ./cookbooks --user sam --server-url http://127.0.0.1:4000 --config /Users/sam/ODI/cuke-chef/.cucumber-chef/vagrant/knife.rb", #<OpenStruct exit_code=0, silence=true>) failed! [1] (ZTK::CommandError)
      features/odi/odi.feature:11:in `* the following cookbooks have been uploaded:'

I think there's some impedance mismatch between librarian-chef and my vagrant/knife.rb. I will dig into this tomorrow. But thanks, this has moved me a long way forward :)

pikesley commented 11 years ago

Oh hold on a moment, I don't have either of the .pem files. That would explain a great deal...

YungSang commented 11 years ago

~~What version of chef-client do you use? I downloaded version 1.1.2 from http://community.opscode.com/cookbooks/chef-client . The latest one dosen't work somehow.~~

Or you can use a blank cookbook for the test like my gist.

YungSang commented 11 years ago

I found an error at cron.rb recipe in the latest chef-client v2.1.8 using foodcritic. line 83; < checksum = Digest::MD5.hexdigest "#{node['fqdn'] or 'unknown-hostname'}" > checksum = Digest::MD5.hexdigest (node['fqdn'] or 'unknown-hostname')

Now works fine with chef-client v2.1.10 and cron v1.2.2

anujbiyani commented 11 years ago

I believe we can expand this issue to include a net-ssh dependency issue. While chef has relaxed their dependency for json from chef requires json (<= 1.6.1, >= 1.4.4) to chef requires json (>= 1.4.4, ~> 1.7.6), vagrant seems to have a dependency on json and net-ssh that conflicts with multiple other gems: vagrant requi res json (~> 1.5.1), net-ssh (~> 2.2.2).

It's worth noting that I'm using cucumber-chef v2.1.0.rc.8.

The full error from my cucumber-chef.log is below, but the short of it is that net-ssh needs to be >= 2.6.5 and json ~> 1.7.6 because of other gems, but vagrant conflicts with both of these.

Here's the full error from my cucumber-chef.log:

---- Begin output of /usr/bin/gem install cucumber-chef -q --no-rdoc --no-ri -v "2.1.0.rc.8" --prerelease ----
STDOUT: 
STDERR: ERROR:  While executing gem ... (Gem::DependencyError)
    Unable to resolve dependencies: net-ssh-multi requires net-ssh (>= 2.1.4); fog requires net-ssh (>= 2.1.3); gherkin requires json (>= 1.4.6); ztk requires net-ssh (>= 0); net-
scp requires net-ssh (>= 1.99.1); chef requires json (>= 1.4.4, ~> 1.7.6), net-ssh (~> 2.6); cucumber requires json (>= 1.4.6); net-sftp requires net-ssh (>= 2.6.5); vagrant requi
res json (~> 1.5.1), net-ssh (~> 2.2.2); net-ssh-gateway requires net-ssh (>= 2.6.5)
        /usr/local/lib/site_ruby/1.8/rubygems/dependency_installer.rb:153:in `gather_dependencies'
        /usr/local/lib/site_ruby/1.8/rubygems/dependency_installer.rb:267:in `install'
        /usr/local/lib/site_ruby/1.8/rubygems/commands/install_command.rb:121:in `execute'
        /usr/local/lib/site_ruby/1.8/rubygems/commands/install_command.rb:115:in `each'
        /usr/local/lib/site_ruby/1.8/rubygems/commands/install_command.rb:115:in `execute'
        /usr/local/lib/site_ruby/1.8/rubygems/command.rb:278:in `invoke'
        /usr/local/lib/site_ruby/1.8/rubygems/command_manager.rb:147:in `process_args'
        /usr/local/lib/site_ruby/1.8/rubygems/command_manager.rb:117:in `run'
        /usr/local/lib/site_ruby/1.8/rubygems/gem_runner.rb:65:in `run'
        /usr/bin/gem:21
---- End output of /usr/bin/gem install cucumber-chef -q --no-rdoc --no-ri -v "2.1.0.rc.8" --prerelease ----

anujbiyani commented 11 years ago

zpatten commented 11 years ago

I feel a Captain Picard face-palm coming on!

zpatten commented 11 years ago

OK; so here's what I have.

I've taken out all the code that calls vagrant internals and removed vagrant as a dep.

You still need vagrant in your path.

I can setup and run tests OK again!

zpatten commented 11 years ago

~~If your using bundler (i.e. Gemfile) with your chef-repo; drop vagrant out of your Gemfile; clean your gemset to ensure vagrant is flushed out; re-bundle; them gem install vagrant separately.~~

If your using bundler; you can point vagrant at this:

git@github.com:jovelabs/vagrant.git

I suggest using this: gem "vagrant", :git => "git@github.com:jovelabs/vagrant.git", :require => false

anujbiyani commented 11 years ago

:+1: cucumber-chef setup ran to completion without an error! Running my test suite now.

zpatten commented 11 years ago

Closing this out now; please reopen if you have the need.

Atalanta / cucumber-chef

Dependency Hell #96