Bundler falls back to legacy mode when you have more than 100 gems in your Gemfile.

[judofyr]

Example Gemfile: https://gist.github.com/whitequark/cfe5e5f70ef22a28cab5 (donated by @whitequark)

https://github.com/bundler/bundler/blob/6423da7150b880a68d8378c200de11f8c0a4b070/lib/bundler/source/rubygems.rb#L229

if idx.size < API_REQUEST_LIMIT && dependency_names.size < API_REQUEST_LIMIT
  # use legacy mode
else
  # use the new API
end

I don't quite understand this code. The comment talks about "non-api sites", but in this example there are no "non-api sites"; there's just good ol' rubygems.org.

[judofyr]

Also, I believe @gnufied had a look at this as well.

[hone]

A non api site in that Gemfile would be any git sources.

[judofyr]

There's not 100 git sources though. There's not over 100 gems in any non-api source in this example.

Currently it takes several minutes (3-6min) to resolve this example Gemfile. If tweak rubygems.rb to use the API for rubygems.org, it takes 30 seconds

[indirect]

Sorry, I'm just not seeing that. First, Bundler uses the API, and second, it resolves instantly as soon as the gemspecs are downloaded. Are you using Bundler 1.3.5?

[judofyr]

Here's what I'm seeing:

$ bundle --version
Bundler version 1.3.5
$ time bundle -V
Updating https://github.com/kossnocorp/jquery-ui-rails.git
Updating git://github.com/neverhood/font-awesome-more-rails.git
Updating git://github.com/javan/whenever.git
Updating git://github.com/whitequark/suwabara.git
Updating git://github.com/evilmartians/liquor.git
Updating git://github.com/conickal/validates_url_format_of.git
Updating git://github.com/evrone/yandex-cleanweb.git
Updating git://github.com/pcreux/rspec-set.git
Fetching from: https://rubygems.org/api/v1/dependencies
HTTP Redirection
Fetching from: https://bundler.rubygems.org/api/v1/dependencies
HTTP Success
Fetching source index from https://rubygems.org/
Resolving dependencies...Fetching from: https://rubygems.org/quick/Marshal.4.8/compass-0.13.alpha.2.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/compass-0.13.alpha.2.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/zendesk_remote_auth-1.2.0.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/zendesk_remote_auth-1.2.0.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/thinking-sphinx-2.0.14.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/thinking-sphinx-2.0.14.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/rb-inotify-0.9.0.gemspec.rz
HTTP Redirection
Fetching from: https://bb-m.rubygems.org/quick/Marshal.4.8/rb-inotify-0.9.0.gemspec.rz
HTTP Success
Fetching from: https://rubygems.org/quick/Marshal.4.8/coffee-rails-3.2.2.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/coffee-rails-3.2.2.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/sass-rails-3.2.6.gemspec.rz
HTTP Redirection
Fetching from: https://bb-m.rubygems.org/quick/Marshal.4.8/sass-rails-3.2.6.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/compass-rails-1.0.3.gemspec.rz
HTTP Redirection
Fetching from: https://bb-m.rubygems.org/quick/Marshal.4.8/compass-rails-1.0.3.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/russian-0.6.0.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/russian-0.6.0.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/hashie-1.2.0.gemspec.rz
HTTP Redirection
Fetching from: https://bb-m.rubygems.org/quick/Marshal.4.8/hashie-1.2.0.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/jquery-rails-cdn-0.2.1.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/jquery-rails-cdn-0.2.1.gemspec.rz
HTTP Success
Fetching from: https://rubygems.org/quick/Marshal.4.8/jquery-rails-2.1.3.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/jquery-rails-2.1.3.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/sidekiq-2.9.0.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/sidekiq-2.9.0.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/chosen-rails-0.9.8.3.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/chosen-rails-0.9.8.3.gemspec.rz
HTTP Success
Fetching from: https://rubygems.org/quick/Marshal.4.8/tinymce-rails-3.5.7.1.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/tinymce-rails-3.5.7.1.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/modernizr-rails-2.6.1.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/modernizr-rails-2.6.1.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/simple_form-2.0.2.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/simple_form-2.0.2.gemspec.rz
HTTP Success
Fetching from: https://rubygems.org/quick/Marshal.4.8/devise-2.2.3.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/devise-2.2.3.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/listen-0.5.3.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/listen-0.5.3.gemspec.rz
HTTP Success
..Fetching from: https://rubygems.org/quick/Marshal.4.8/rails-3.2.13.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/rails-3.2.13.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/spring-0.0.8.gemspec.rz
HTTP Redirection
Fetching from: https://s3.amazonaws.com/production.s3.rubygems.org/quick/Marshal.4.8/spring-0.0.8.gemspec.rz
HTTP Success
.Fetching from: https://rubygems.org/quick/Marshal.4.8/streamio-ffmpeg-0.9.0.gemspec.rz
^C
real    0m57.786s
user    0m24.428s
sys     0m0.450s

[judofyr]

idx.size is 0, but dependency_names.size is 125:

["rails", "russian", "unicorn", "pg", "activerecord-postgres-hstore", "thinking-sphinx", "rack-utf8_sanitizer", "slim", "coffee-rails", "sass-rails", "sass", "compass-rails", "compass", "therubyracer", "uglifier", "jquery-rails", "turbo-sprockets-rails3", "jquery-rails-cdn", "tinymce-rails", "tinymce-rails-langs", "jquery-ui-rails", "chosen-rails", "modernizr-rails", "font-awesome-more-rails", "oj", "rollbar", "devise", "devise-encryptable", "devise-async", "cancan", "sidekiq", "sidekiq-failures", "whenever", "sinatra", "mailcatcher", "friendly_id", "unicode", "babosa", "dalli", "dynamic_form", "simple_form", "suwabara", "acts_as_paranoid", "routing_concerns", "domain_name", "htmlentities", "kaminari", "liquor", "mime-types", "mechanize", "multipart-post", "net-scp", "net-ssh", "newrelic_rpm", "nokogiri", "paper_trail", "rack-www", "rails_config", "rest-client", "sanitize", "hashie", "streamio-ffmpeg", "validates_url_format_of", "active_model_serializers", "will_paginate", "oauth", "oauth2", "fb_graph", "tumblr_client", "soundcloud", "gibbon", "zendesk_api", "zendesk_remote_auth", "youtube_it", "yandex_cleanweb", "net-dns", "whois", "rspec-rails", "pry", "pry-rails", "ruby-prof", "better_errors", "binding_of_caller", "capistrano", "capistrano-ext", "rvm-capistrano", "capistrano-detect-migrations", "capistrano-campfire", "tinder", "capistrano_colors", "quiet_assets", "spring", "listen", "guard", "guard-rspec", "guard-konacha", "rb-fsevent", "rb-inotify", "libnotify", "simplecov", "simplecov-summary", "database_cleaner", "rspec-set", "fuubar", "factory_girl_rails", "shoulda", "rake", "minitest", "minitest-spec-context", "activerecord", "rdiscount", "sqlite3", "yard", "racc", "kramdown", "rspec", "mini_magick", "rack", "coveralls", "chronic", "activesupport", "mocha", "railties", "tzinfo", "json"]

[gnufied]

Bundler downloads dependency list in two steps:

1. Use bundler specific API to download dependency tuples and build a index with this data. This data is more than enough to resolve the dependencies.
2. Bundler runs recursive backtracking algorithm on this index to determine, which dependencies are final.
3. Bundler downloads complete gemspec of those final dependencies, builds an index and proceeds to install them.

This Gemfile with 125 direct dependencies does trigger legacy download and bundler is forced to download full index and hence the problem. I am not sure what is the fix, but this indeed is bit of a problem.

@indirect @judofyr @hone

[gnufied]

I am going to mark this one duplicate of #2448 . Probably worth finding a middle ground here.