rubygems / bundler-api

Bundler 1.1+ API
46 stars 39 forks source link

Bundler freezes on dependencies #21

Closed brandonhilkert closed 11 years ago

brandonhilkert commented 11 years ago

Sorry if this isn't the right place for this. But I saw this issue ( https://github.com/rubygems/bundler-api/issues/17) and it resembled my issue, so I'll give it a shot.

I'm a contributor to https://github.com/mperham/sidekiq and a few days ago the Travis CI tests started erroring out (ie. https://travis-ci.org/mperham/sidekiq/jobs/4360052). As you can see, it makes the call to bundle install, prints the Fetching gem metadata from http://rubygems.org/ twice, and then freezes. Here's the verbose output:

[11:48:42] bhilkert [~/Dropbox/code/sidekiq] (master) $ bundle outdated --verbose
Fetching gem metadata from http://rubygems.org/
Query List: ["sidekiq", "minitest", "sinatra", "slim", "rake", "actionmailer", "activerecord", "celluloid", "sqlite3", "simplecov", "pry", "shotgun", "redis", "redis-namespace", "connection_pool", "multi_json"]
Query Gemcutter Dependency Endpoint API: sidekiq minitest sinatra slim rake actionmailer activerecord celluloid sqlite3 simplecov pry shotgun redis redis-namespace connection_pool multi_json
Fetching from: http://rubygems.org/api/v1/dependencies?gems=sidekiq,minitest,sinatra,slim,rake,actionmailer,activerecord,celluloid,sqlite3,simplecov,pry,shotgun,redis,redis-namespace,connection_pool,multi_json
HTTP Success
Query List: ["mail", "actionpack", "text-format", "tzinfo", "arel", "activemodel", "activesupport", "facter", "timers", "hoe", "win32console", "method_source", "slop", "coderay", "spoon", "ruby_parser", "yard", "rspec", "rack", "launchy", "compass", "sass", "sprockets-sass", "simplecov-html", "tilt", "rack-protection", "mongrel", "rack-test", "temple", "escape_utils", "ffi"]
Query Gemcutter Dependency Endpoint API: mail actionpack text-format tzinfo arel activemodel activesupport facter timers hoe win32console method_source slop coderay spoon ruby_parser yard rspec rack launchy compass sass sprockets-sass simplecov-html tilt rack-protection mongrel rack-test temple escape_utils ffi
Fetching from: http://rubygems.org/api/v1/dependencies?gems=mail,actionpack,text-format,tzinfo,arel,activemodel,activesupport,facter,timers,hoe,win32console,method_source,slop,coderay,spoon,ruby_parser,yard,rspec,rack,launchy,compass,sass,sprockets-sass,simplecov-html,tilt,rack-protection,mongrel,rack-test,temple,escape_utils,ffi
HTTP Success
Query List: ["erubis", "sprockets", "journey", "builder", "rack-cache", "rack-mount", "i18n", "bcrypt-ruby", "memcache-client", "haml", "chunky_png", "fssm", "bones", "RubyInline", "rubyforge", "gemcutter", "configuration", "addressable", "treetop", "mime-types", "tlsmail", "daemons", "fastthread", "gem_plugin", "cgi_multipart_eof_fix", "test-spec", "camping", "fcgi", "ruby-openid", "thin", "rspec-mocks", "rspec-expectations", "rspec-core", "cucumber", "diff-lcs", "syntax", "spicycode-rcov", "sexp_processor", "ParseTree", "sprockets-helpers", "text-hyphen"]
Query Gemcutter Dependency Endpoint API: erubis sprockets journey builder rack-cache rack-mount i18n bcrypt-ruby memcache-client haml chunky_png fssm bones RubyInline rubyforge gemcutter configuration addressable treetop mime-types tlsmail daemons fastthread gem_plugin cgi_multipart_eof_fix test-spec camping fcgi ruby-openid thin rspec-mocks rspec-expectations rspec-core cucumber diff-lcs syntax spicycode-rcov sexp_processor ParseTree sprockets-helpers text-hyphen
Fetching from: http://rubygems.org/api/v1/dependencies?gems=erubis,sprockets,journey,builder,rack-cache,rack-mount,i18n,bcrypt-ruby,memcache-client,haml,chunky_png,fssm,bones,RubyInline,rubyforge,gemcutter,configuration,addressable,treetop,mime-types,tlsmail,daemons,fastthread,gem_plugin,cgi_multipart_eof_fix,test-spec,camping,fcgi,ruby-openid,thin,rspec-mocks,rspec-expectations,rspec-core,cucumber,diff-lcs,syntax,spicycode-rcov,sexp_processor,ParseTree,sprockets-helpers,text-hyphen
HTTP Success
Query List: ["little-plugger", "loquacious", "bones-git", "bones-extras", "markaby", "metaid", "mab", "term-ansicolor", "polyglot", "gherkin", "json_pure", "json", "abstract", "net-scp", "maruku", "ZenTest", "archive-tar-minitar", "nokogiri", "rcov", "SexpProcessor", "multimap", "ruby-yadis", "hike", "flexmock", "mocha", "eventmachine", "eventmachine-le", "http_parser.rb", "preforker", "facets"]
Query Gemcutter Dependency Endpoint API: little-plugger loquacious bones-git bones-extras markaby metaid mab term-ansicolor polyglot gherkin json_pure json abstract net-scp maruku ZenTest archive-tar-minitar nokogiri rcov SexpProcessor multimap ruby-yadis hike flexmock mocha eventmachine eventmachine-le http_parser.rb preforker facets
Fetching from: http://rubygems.org/api/v1/dependencies?gems=little-plugger,loquacious,bones-git,bones-extras,markaby,metaid,mab,term-ansicolor,polyglot,gherkin,json_pure,json,abstract,net-scp,maruku,ZenTest,archive-tar-minitar,nokogiri,rcov,SexpProcessor,multimap,ruby-yadis,hike,flexmock,mocha,eventmachine,eventmachine-le,http_parser.rb,preforker,facets
HTTP Success
Query List: ["bones-rcov", "bones-rubyforge", "bones-rspec", "bones-zentest", "git", "trollop", "spruz", "metaclass", "net-ssh", "racc", "tenderlove-frex", "rake-compiler", "rexical", "weakling"]
Query Gemcutter Dependency Endpoint API: bones-rcov bones-rubyforge bones-rspec bones-zentest git trollop spruz metaclass net-ssh racc tenderlove-frex rake-compiler rexical weakling
Fetching from: http://rubygems.org/api/v1/dependencies?gems=bones-rcov,bones-rubyforge,bones-rspec,bones-zentest,git,trollop,spruz,metaclass,net-ssh,racc,tenderlove-frex,rake-compiler,rexical,weakling
HTTP Success
Query List: ["needle", "jruby-pageant", "echoe"]
Query Gemcutter Dependency Endpoint API: needle jruby-pageant echoe
Fetching from: http://rubygems.org/api/v1/dependencies?gems=needle,jruby-pageant,echoe
HTTP Success
Query List: ["highline", "allison", "rdoc"]
Query Gemcutter Dependency Endpoint API: highline allison rdoc
Fetching from: http://rubygems.org/api/v1/dependencies?gems=highline,allison,rdoc
HTTP Success
Query List: ["termios"]
Query Gemcutter Dependency Endpoint API: termios
Fetching from: http://rubygems.org/api/v1/dependencies?gems=termios
HTTP Success
Query List: []
Unmet Dependencies: ["tenderlove-frex", "spicycode-rcov"]
Fetching gem metadata from http://rubygems.org/
Query List: ["tenderlove-frex", "spicycode-rcov"]
Query Gemcutter Dependency Endpoint API: tenderlove-frex spicycode-rcov
Fetching from: http://rubygems.org/api/v1/dependencies?gems=tenderlove-frex,spicycode-rcov
HTTP Success
Query List: []

I should also mention that I didn't originally notice this b/c I had all the dependencies installed on my machine already, so bundle install wasn't doing any more. Only when I ran bundle outdated did it actually go out and hit this. https://github.com/mperham doesn't have this issue. He is using rvm. I have using Rbenv, but I fail to understand why that would change anything.

I also tried to update bundler, but to no avail:

[12:03:53] bhilkert [~/Dropbox/code/sidekiq] (master) $ gem update bundle
Updating installed gems
Updating bundler
Fetching: bundler-1.2.3.gem (100%)
Successfully installed bundler-1.2.3
Gems updated: bundler
Installing ri documentation for bundler-1.2.3...
Installing RDoc documentation for bundler-1.2.3...

I found a few other cases involving tenderlove-frex:

https://groups.google.com/forum/?fromgroups=#!topic/rubygems-org/sgpO6ed_kAM

http://stackoverflow.com/questions/13443332/bundle-update-stuck

http://proxy.obd2tool.com/index.php?url=141L19y91q1HyOyr0I1CyX1s0UyGyFyMyb1v121t1lyd0Kyj1tyN1xya

http://iqwen.net/question/72831

Any advice?

indirect commented 11 years ago

Tenderlove-frex is a bit of a red herring -- it only shows up at the end because its not actually available from rubygems.org. (It was a github gem.)

What's actually happening is that Bundler is trying to resolve the sidekiq Gemfile, and there are so many possibilities that it's taking longer than the Travis timeout to run.

Options include copying a Gemfile.lock to Travis before bundle install runs, or adjusting the sidekiq dependencies to be more specific so that Bundler has less problem space to cover.

On Jan 27, 2013, at 9:31 AM, Brandon Hilkert notifications@github.com wrote:

Sorry if this isn't the right place for this. But I saw this issue ( #17) and it resembled my issue, so I'll give it a shot.

I'm a contributor to https://github.com/mperham/sidekiq and a few days ago the Travis CI tests started erroring out (ie. https://travis-ci.org/mperham/sidekiq/jobs/4360052). As you can see, it makes the call to bundle install, prints the Fetching gem metadata from http://rubygems.org/ twice, and then freezes. Here's the verbose output:

[11:48:42] bhilkert ~/Dropbox/code/sidekiq $ bundle outdated --verbose Fetching gem metadata from http://rubygems.org/ Query List: ["sidekiq", "minitest", "sinatra", "slim", "rake", "actionmailer", "activerecord", "celluloid", "sqlite3", "simplecov", "pry", "shotgun", "redis", "redis-namespace", "connection_pool", "multi_json"] Query Gemcutter Dependency Endpoint API: sidekiq minitest sinatra slim rake actionmailer activerecord celluloid sqlite3 simplecov pry shotgun redis redis-namespace connection_pool multi_json Fetching from: http://rubygems.org/api/v1/dependencies?gems=sidekiq,minitest,sinatra,slim,rake,actionmailer,activerecord,celluloid,sqlite3,simplecov,pry,shotgun,redis,redis-namespace,connection_pool,multi_json HTTP Success Query List: ["mail", "actionpack", "text-format", "tzinfo", "arel", "activemodel", "activesupport", "facter", "timers", "hoe", "win32console", "method_source", "slop", "coderay", "spoon", "ruby_parser", "yard", "rspec", "rack", "launchy", "compass", "sass", "sprockets-sass", "simplecov-html", "tilt", "rack-protection", "mongrel", "rack-test", "temple", "escape_utils", "ffi"] Query Gemcutter Dependency Endpoint API: mail actionpack text-format tzinfo arel activemodel activesupport facter timers hoe win32console method_source slop coderay spoon ruby_parser yard rspec rack launchy compass sass sprockets-sass simplecov-html tilt rack-protection mongrel rack-test temple escape_utils ffi Fetching from: http://rubygems.org/api/v1/dependencies?gems=mail,actionpack,text-format,tzinfo,arel,activemodel,activesupport,facter,timers,hoe,win32console,method_source,slop,coderay,spoon,ruby_parser,yard,rspec,rack,launchy,compass,sass,sprockets-sass,simplecov-html,tilt,rack-protection,mongrel,rack-test,temple,escape_utils,ffi HTTP Success Query List: ["erubis", "sprockets", "journey", "builder", "rack-cache", "rack-mount", "i18n", "bcrypt-ruby", "memcache-client", "haml", "chunky_png", "fssm", "bones", "RubyInline", "rubyforge", "gemcutter", "configuration", "addressable", "treetop", "mime-types", "tlsmail", "daemons", "fastthread", "gem_plugin", "cgi_multipart_eof_fix", "test-spec", "camping", "fcgi", "ruby-openid", "thin", "rspec-mocks", "rspec-expectations", "rspec-core", "cucumber", "diff-lcs", "syntax", "spicycode-rcov", "sexp_processor", "ParseTree", "sprockets-helpers", "text-hyphen"] Query Gemcutter Dependency Endpoint API: erubis sprockets journey builder rack-cache rack-mount i18n bcrypt-ruby memcache-client haml chunky_png fssm bones RubyInline rubyforge gemcutter configuration addressable treetop mime-types tlsmail daemons fastthread gem_plugin cgi_multipart_eof_fix test-spec camping fcgi ruby-openid thin rspec-mocks rspec-expectations rspec-core cucumber diff-lcs syntax spicycode-rcov sexp_processor ParseTree sprockets-helpers text-hyphen Fetching from: http://rubygems.org/api/v1/dependencies?gems=erubis,sprockets,journey,builder,rack-cache,rack-mount,i18n,bcrypt-ruby,memcache-client,haml,chunky_png,fssm,bones,RubyInline,rubyforge,gemcutter,configuration,addressable,treetop,mime-types,tlsmail,daemons,fastthread,gem_plugin,cgi_multipart_eof_fix,test-spec,camping,fcgi,ruby-openid,thin,rspec-mocks,rspec-expectations,rspec-core,cucumber,diff-lcs,syntax,spicycode-rcov,sexp_processor,ParseTree,sprockets-helpers,text-hyphen HTTP Success Query List: ["little-plugger", "loquacious", "bones-git", "bones-extras", "markaby", "metaid", "mab", "term-ansicolor", "polyglot", "gherkin", "json_pure", "json", "abstract", "net-scp", "maruku", "ZenTest", "archive-tar-minitar", "nokogiri", "rcov", "SexpProcessor", "multimap", "ruby-yadis", "hike", "flexmock", "mocha", "eventmachine", "eventmachine-le", "http_parser.rb", "preforker", "facets"] Query Gemcutter Dependency Endpoint API: little-plugger loquacious bones-git bones-extras markaby metaid mab term-ansicolor polyglot gherkin json_pure json abstract net-scp maruku ZenTest archive-tar-minitar nokogiri rcov SexpProcessor multimap ruby-yadis hike flexmock mocha eventmachine eventmachine-le http_parser.rb preforker facets Fetching from: http://rubygems.org/api/v1/dependencies?gems=little-plugger,loquacious,bones-git,bones-extras,markaby,metaid,mab,term-ansicolor,polyglot,gherkin,json_pure,json,abstract,net-scp,maruku,ZenTest,archive-tar-minitar,nokogiri,rcov,SexpProcessor,multimap,ruby-yadis,hike,flexmock,mocha,eventmachine,eventmachine-le,http_parser.rb,preforker,facets HTTP Success Query List: ["bones-rcov", "bones-rubyforge", "bones-rspec", "bones-zentest", "git", "trollop", "spruz", "metaclass", "net-ssh", "racc", "tenderlove-frex", "rake-compiler", "rexical", "weakling"] Query Gemcutter Dependency Endpoint API: bones-rcov bones-rubyforge bones-rspec bones-zentest git trollop spruz metaclass net-ssh racc tenderlove-frex rake-compiler rexical weakling Fetching from: http://rubygems.org/api/v1/dependencies?gems=bones-rcov,bones-rubyforge,bones-rspec,bones-zentest,git,trollop,spruz,metaclass,net-ssh,racc,tenderlove-frex,rake-compiler,rexical,weakling HTTP Success Query List: ["needle", "jruby-pageant", "echoe"] Query Gemcutter Dependency Endpoint API: needle jruby-pageant echoe Fetching from: http://rubygems.org/api/v1/dependencies?gems=needle,jruby-pageant,echoe HTTP Success Query List: ["highline", "allison", "rdoc"] Query Gemcutter Dependency Endpoint API: highline allison rdoc Fetching from: http://rubygems.org/api/v1/dependencies?gems=highline,allison,rdoc HTTP Success Query List: ["termios"] Query Gemcutter Dependency Endpoint API: termios Fetching from: http://rubygems.org/api/v1/dependencies?gems=termios HTTP Success Query List: [] Unmet Dependencies: ["tenderlove-frex", "spicycode-rcov"] Fetching gem metadata from http://rubygems.org/ Query List: ["tenderlove-frex", "spicycode-rcov"] Query Gemcutter Dependency Endpoint API: tenderlove-frex spicycode-rcov Fetching from: http://rubygems.org/api/v1/dependencies?gems=tenderlove-frex,spicycode-rcov HTTP Success Query List: [] I should also mention that I didn't originally notice this b/c I had all the dependencies installed on my machine already, so bundle install wasn't doing any more. Only when I ran bundle outdated did it actually go out and hit this. https://github.com/mperham doesn't have this issue. He is using rvm. I have using Rbenv, but I fail to understand why that would change anything.

I also tried to update bundler, but to no avail:

[12:03:53] bhilkert ~/Dropbox/code/sidekiq $ gem update bundle Updating installed gems Updating bundler Fetching: bundler-1.2.3.gem (100%) Successfully installed bundler-1.2.3 Gems updated: bundler Installing ri documentation for bundler-1.2.3... Installing RDoc documentation for bundler-1.2.3... I found a few other cases involving tenderlove-frex:

https://groups.google.com/forum/?fromgroups=#!topic/rubygems-org/sgpO6ed_kAM

http://stackoverflow.com/questions/13443332/bundle-update-stuck

http://proxy.obd2tool.com/index.php?url=141L19y91q1HyOyr0I1CyX1s0UyGyFyMyb1v121t1lyd0Kyj1tyN1xya

http://iqwen.net/question/72831

Any advice?

\ Reply to this email directly or view it on GitHub.

brandonhilkert commented 11 years ago

Got it. I isolated it down to the gems 'pry' and 'shotgun'. Since they're isolated to the dev ENV, I'm thinking I just exclude that ENV on the bundle command when Travis builds.

Thanks for the ideas.

brandonhilkert commented 11 years ago

I changed the Travis config and that didn't fix it. Is the fact that something relies on "frex" the problem?

indirect commented 11 years ago

@brandonhilkert that's not the issue — the most likely reason is simply that there are too many gems for Bundler to check every possible combination in a reasonable amount of time.

indirect commented 11 years ago

@brandonhilkert so I spent a few minutes playing with this... here's how to make bundle install resolve instantly. Add this line to the Gemfile: gem 'rack', '~> 1.2.0'. That's it.

It turns out that Rack 1.5.1 is getting activated, and actionpack 3 depends on Rack 1.2. This conflict means that Bundler is trying every possible version of... a lot of gems. :(

brandonhilkert commented 11 years ago

That did it! Thank you so much.


http://brandonhilkert.com

On Fri, Feb 1, 2013 at 3:26 AM, André Arko notifications@github.com wrote:

@brandonhilkert https://github.com/brandonhilkert so I spent a few minutes playing with this... here's how to make bundle install resolve instantly. Add this line to the Gemfile: gem 'rack', '~> 1.2.0'. That's it.

It turns out that Rack 1.5.1 is getting activated, and actionpack 3 depends on Rack 1.2. This conflict means that Bundler is trying every possible version of... a lot of gems. :(

— Reply to this email directly or view it on GitHubhttps://github.com/rubygems/bundler-api/issues/21#issuecomment-12984912.