Can't install railties 4.1.0.rc2 with rspec-rails 3.0.0.beta2 with no good reason

[rosenfeld]

Gemfile:

source 'https://rubygems.org'

gem 'railties', '4.1.0.rc2'
gem 'rspec-rails', '3.0.0.beta2'

bundle install output:

Fetching gem metadata from https://rubygems.org/.........
Fetching additional metadata from https://rubygems.org/..
Resolving dependencies...
Bundler could not find compatible versions for gem "activesupport":
  In Gemfile:
    rspec-rails (= 3.0.0.beta2) ruby depends on
      activesupport (= 3.0.0) ruby

    railties (= 4.1.0.rc2) ruby depends on
      activesupport (4.1.0.rc2)

[rosenfeld]

Forgot to mention that rspec-rails depends on activesupport >= 3.0, not equal to 3.0.0 like reported by bundle install:

https://github.com/rspec/rspec-rails/blob/v3.0.0.beta2/rspec-rails.gemspec#L30

[indirect]

4.1.0.rc2 is a prerelease, which means it does not match >= 3.0. It would match >= 3.0.a, though. That is an extremely good reason.

You need to either wait until 4.1 final is out, ask rspec to change the version requirement, or fork the rspec repo, change the requirement, and then add your forked git repo to your gemfile.

On Mar 26, 2014, at 7:47 AM, Rodrigo Rosenfeld Rosas notifications@github.com wrote:

Forgot to mention that rspec-rails depends on activesupport >= 3.0, not equal to 3.0.0 like reported by bundle install:

https://github.com/rspec/rspec-rails/blob/v3.0.0.beta2/rspec-rails.gemspec#L30

— Reply to this email directly or view it on GitHub.

[rosenfeld]

Great, this is puzzling me for several months now and I always had to fiddle with Gemfile.lock to get things to work when that happens... At least now I understand the reason. Is that behavior documented somewhere?

[rosenfeld]

found it here: http://guides.rubygems.org/patterns/#pessimistic_version_constraint

"If you want to allow prereleases and regular releases use a compound requirement: ..."

[indirect]

It’s also mentioned at the bottom of the page where it talks about prerelease versioning:

http://guides.rubygems.org/patterns/#prerelease_gems

Glad you found it. :)

On Mar 26, 2014, at 8:47 AM, Rodrigo Rosenfeld Rosas notifications@github.com wrote:

found it here: http://guides.rubygems.org/patterns/#pessimistic_version_constraint

"If you want to allow prereleases and regular releases use a compound requirement: ..."

— Reply to this email directly or view it on GitHub.

[rosenfeld]

Actually, the documentation in prerelease gems says nothing about how >= works with regards to pre-release gems, which I find a bit non intuitive. I think it's okay to install latest stable release when a >= requirement is set, but I don't think it should prevent pre releases from working when they are specified elsewhere. It seems intuitive to me think that >= 3.0 would satisfy 3.0.1.beta1 for instance. And unless the beta release is required somewhere in the gems dependency bundler shouldn't select it, but the latest released gem that satisfies the >= 3.0 criteria instead.

I'm confused. Suppose a gem depends on gem-a, which depends on gem-b >= '3.0.0.a'. Questions:

1 - if latest gem-b version (including pre releases) is '3.0.0', will it match the dependency? 2 - if you depend on gem-a (no explicit version) and nothing else, and gem-b is available on 3.0.0 and 3.0.1.beta1, which gem-b version will be installed? 3.0.0? 3.0.1.beta1? Is there anyway I could enforce 3.0.0 to be installed in such situation, when I didn't explicitly asked the pre release to be installed?

[indirect]

I'm not sure of the answers. You can check yourself using IRB with Gem::Requirement.new(">= 3.0.0.a").satisfied_by? Gem::Version.new("3.0.0").

On Wed, Mar 26, 2014 at 9:08 AM, Rodrigo Rosenfeld Rosas notifications@github.com wrote:

Actually, the documentation in prerelease gems says nothing about how >= works with regards to pre-release gems, which I find a bit non intuitive. I think it's okay to install latest stable release when a >= requirement is set, but I don't think it should prevent pre releases from working when they are specified elsewhere. It seems intuitive to me think that >= 3.0 would satisfy 3.0.1.beta1 for instance. And unless the beta release is required somewhere in the gems dependency bundler shouldn't select it, but the latest released gem that satisfies the >= 3.0 criteria instead. I'm confused. Suppose a gem depends on gem-a, which depends on gem-b >= '3.0.0.a'. Questions: 1 - if latest gem-b version (including pre releases) is '3.0.0', will it match the dependency?

2 - if you depend on gem-a (no explicit version) and nothing else, and gem-b is available on 3.0.0 and 3.0.1.beta1, which gem-b version will be installed? 3.0.0? 3.0.1.beta1? Is there anyway I could enforce 3.0.0 to be installed in such situation, when I didn't explicitly asked the pre release to be installed?

Reply to this email directly or view it on GitHub: https://github.com/bundler/bundler/issues/2941#issuecomment-38703266

[rosenfeld]

Great, good to know. By the way, 3.0.0 satisfies >= 3.0.0.a. Any trick to test about Bundler resolution without actually having to call bundle install?

[indirect]

If you already have all the gems installed, you can run bundle install --local or run Bundler.setup in IRB. If the gems aren't already installed, you have to bundle install to fetch all the version info and look for versions that will work.

On Wed, Mar 26, 2014 at 11:36 AM, Rodrigo Rosenfeld Rosas notifications@github.com wrote:

Great, good to know. By the way, 3.0.0 satisfies >= 3.0.0.a. Any trick to test about Bundler resolution without actually having to call bundle install?

Reply to this email directly or view it on GitHub: https://github.com/bundler/bundler/issues/2941#issuecomment-38722529

[rosenfeld]

Ok, I was wondering if there was an easier way to do that, but bundle install --local seems to be the fastest alternative I know. I tried a new Gemfile (no existing Gemfile.lock) with this content:

source 'https://rubygems.org'
gem 'rspec-rails', github: 'rosenfeld/rspec-rails', branch: 'patch-1'

As I suspected (and was afraid of), railties 4.1.0.rc2 was pick instead of the latest stable one. Is there anyway to tell Bundler that 4.1.0.rc2 would satisfy a >= 3.0.0 but that Bundler should prefer stable versions when no specific pre-release version is specified?

From the Gemfile above, I'd prefer the latest stable Railties to be installed.

[rosenfeld]

Would you mind to explain why the following works:

source 'https://rubygems.org'

gem 'rails', '4.1.0.rc2'

group :development, :test do
  gem 'rspec-rails', '>= 3.0.0.beta2'
end

while this doesn't?

source 'https://rubygems.org'

gem 'railties', '4.1.0.rc2'

group :development, :test do
  gem 'rspec-rails', '>= 3.0.0.beta2'
end

I've only replaced 'rails' with 'railties'. Why does this change will make any difference to Bundler?

What's even more strange is that if I run bundle install with rails and then replace it with railties while keeping the Gemfile.lock, it works. But if I remove the Gemfile.lock it stops working.

[indirect]

because rails is a gem that depends on railties, and rspec-rails is a gem that depends on railties. so you're probably removing a conflicting requirement with that change.

On Wed, Mar 26, 2014 at 6:46 PM, Rodrigo Rosenfeld Rosas notifications@github.com wrote:

Would you mind to explain why the following works:

source 'https://rubygems.org'
gem 'rails', '4.1.0.rc2'
group :development, :test do
  gem 'rspec-rails', '>= 3.0.0.beta2'
end

while this doesn't?

source 'https://rubygems.org'
gem 'railties', '4.1.0.rc2'
group :development, :test do
  gem 'rspec-rails', '>= 3.0.0.beta2'
end

I've only replaced 'rails' with 'railties'. Why does this change will make any difference to Bundler?

What's even more strange is that if I run bundle install with rails and then replace it with railties while keeping the Gemfile.lock, it works. But if I remove the Gemfile.lock it stops working.

Reply to this email directly or view it on GitHub: https://github.com/bundler/bundler/issues/2941#issuecomment-38761227

[Who828]

Let me give a detail explanation for this particular case,

The conflicting pieces are

Bundler could not find compatible versions for gem "activesupport":
  In Gemfile:
    rspec-rails (= 3.0.0.beta2) ruby depends on
      activemodel (>= 3.0) ruby depends on
        activesupport (= 3.0.0) ruby

    railties (= 4.1.0.rc2) ruby depends on
      actionpack (= 4.1.0.rc2) ruby depends on
        activesupport (4.1.0.rc2)

so here rspec-rails relies on activemodel with constraint (>= 3.0), so we have a search method in our resolver which dutifully give us gem versions which are compatible with activated gems so far.

Also in this case because the constraint is (>= 3.0) and not (>= 3.0.a) it will only give us stable versions, the latest stable version is 4.0.4 and now that activemodel gem has dependency on activesupport in form (= 4.0.4).

Note : Bundler will start from 4.0.4 and will go till 3.0.0 (because we have >= 3.0) for activemodel and so it will try activesupport from 4.0.4 to 3.0 as well.

but railties has dependency on actionpack(= 4.1.0.rc2) and it has another dependency on activesupport with constraint (= 4.1.0.rc2) so this can't be resolved.

However, you saw it resolved in case of replace railties with rails. Why did this happen? because Rails is installing activemodel(=4.1.0.rc2) before we even reach the dependency tree of rspec-rails. So when we actually walk to rspec-rails dependency requirement of activemodel(>= 3.0), the activemodel (= 4.1.0.rc2) is already installed.

Now in this case bundler does something like, current.requirement.satisfied_by?(existing.version) which is essentially

Gem::Requirement.new(">= 3.0.0").satisfied_by? Gem::Version.new("4.1.0.rc2")

which is true in this case and thus it goes on to resolve other things and everything works out fine.

So a one work around is to explicitly mention activemodel in your Gemfile, for example

gem 'activemodel', '4.1.0.rc2'

so in this case activemodel will be installed before we walk down the dependency tree.

Now why don't we fetch pre-release versions for dependency in our search method ? Because they are not stable. Fetching pre-release versions of gem for dependency could lead to bugs which would be very hard to debug and you can't blame the gem author in that case because its a pre-release version of the gem.

Think what the impact would be if you had a critical app running on production and it failed because one of your gem had (>= 3.0) requirement and it had installed a pre-release version of the dependent gem.

This is why we recommend to explicitly mention pre-release gems in the gemspec file, it might be painful but it won't come back bite you on when using bundler to resolve files in critical env.

[rosenfeld]

Hi Smit, thank you for your explanation. @cupakromer mentioned recently that Bundler does actually install the gems on his machine: https://github.com/rspec/rspec-rails/pull/972#issuecomment-38761818

So, I still think there's something on Bundler that could be improved so that it would work consistently the same among different environments. Also, calling bundle with --local also seems to make it work:

rm Gemfile.lock
09:14:56 rodrigo@rodrigo:~/tmp/example-rspec-rails
$ bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching additional metadata from https://rubygems.org/..
Resolving dependencies...
Bundler could not find compatible versions for gem "activesupport":
  In Gemfile:
    rspec-rails (= 3.0.0.beta2) ruby depends on
      activesupport (= 3.0.0) ruby

    railties (= 4.1.0.rc2) ruby depends on
      activesupport (4.1.0.rc2)
09:15:13 rodrigo@rodrigo:~/tmp/example-rspec-rails
$ bundle --local
Resolving dependencies...
Using i18n (0.6.9)
Using rake (10.2.1)
Using json (1.8.1)
Using builder (3.2.2)
Using minitest (5.3.1)
Using atomic (1.1.16)
Using erubis (2.7.0)
Using rack (1.5.2)
Using diff-lcs (1.2.5)
Using thor (0.19.1)
Using rspec-support (3.0.0.beta2)
Using bundler (1.5.3)
Using rack-test (0.6.2)
Using rspec-expectations (3.0.0.beta2)
Using thread_safe (0.3.1)
Using rspec-core (3.0.0.beta2)
Using rspec-mocks (3.0.0.beta2)
Using rspec-collection_matchers (0.0.3)
Using tzinfo (1.1.0)
Using activesupport (4.1.0.rc2)
Using actionview (4.1.0.rc2)
Using activemodel (4.1.0.rc2)
Using actionpack (4.1.0.rc2)
Using railties (4.1.0.rc2)
Using rspec-rails (3.0.0.beta2)
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.

If it won't resolve it automatically, it should at least point us to the right correct on its error messages. Here's the message we see:

    rspec-rails (= 3.0.0.beta2) ruby depends on
      activesupport (= 3.0.0) ruby

But this is misleading and not true actually. What happens, if I understand correctly, is that rspec-rails depends on activemodel, which depends on a specific version of activesupport which conflicts with the version required by railties. Isn't that correct?

So, Bundler shouldn't report that rspec-rails (=3.0.0.beta2) depends on activesupport (=3.0.0) because this is clearly not true.

For now, I'll suggest rspec-rails to drop the dependency on activemodel because it doesn't actually depend on it from what I've seen in the codebase so far. I'll re-check and create a PR there, but that's a separate issue from this one.

[Who828]

@rosenfeld Bundler 1.6.0.RC2 has better conflict message, it will show like this

Bundler could not find compatible versions for gem "activesupport":
  In Gemfile:
    rspec-rails (= 3.0.0.beta2) ruby depends on
      activemodel (>= 3.0) ruby depends on
        activesupport (= 3.0.0) ruby

    railties (= 4.1.0.rc2) ruby depends on
      actionpack (= 4.1.0.rc2) ruby depends on
        activesupport (4.1.0.rc2)

It will be in Bundler 1.6 final which is coming out soon.

[rosenfeld]

But the message is still misleading:

activemodel (>= 3.0) ruby depends on
        activesupport (= 3.0.0) ruby

This is still not true.

[Who828]

It is true. Also in this case because the constraint is (>= 3.0) and not (>= 3.0.a) it will only give us stable versions, the latest stable version is 4.0.4 and now that activemodel gem has dependency on activesupport in form (= 4.0.4).

Note : Bundler will start from 4.0.4 and will go till 3.0.0 (because we have >= 3.0) for activemodel and so it will try activesupport from 4.0.4 to 3.0 as well. so it showing

activemodel (>= 3.0) ruby depends on
        activesupport (= 3.0.0) ruby

Which is the last version it tried before it ran out of different gem versions to try.

[rosenfeld]

Exactly, but this is only clear to people who knows how Bundler works behind the scene. The message states that activemodel >=3 depends on activesupport 3.0.0 which is not really true. It's just the last version Bundler has tried for activesupport due to its algorithm.

[rosenfeld]

Maybe it could report something like:

activemodel (>= 3.0) ruby depends on
        activesupport (tried final releases: 3.0.0, 3.0.1, ..., 4.0.2) ruby

With all versions shown actually...

[Who828]

First of all Bundler algorithm is deterministic, if it tells you there is a conflict based on the information it has then it will give you the same conflict no matter how many times you run it.

In your case bundler --local works because its taking information from cached folder and for @cupakromer it's working because of the Gemfile.lock (as I don't see the resolving phase in his logs) the order of resolution is different.

Now about the conflict error message, there are good reasons why we don't show all the versions.

One is because it might bloat your entire terminal with gem versions depending on the number of gem versions Bundler tried (that depends on the gem dependency constraint).

Two we don't know which gem is going be a conflict, so we have to store all of the versions in a hashmap. Now Gemfiles with 80-100 top level gems is a common place for a mature project, the total amount of gems including all of its dependency might reach to 200-300 each with 20-30 versions more or less depending on the constraints(that's a big hashmap). This information is completely wasted if there are no conflicts for example and it makes Bundler use lot of memory with very little benefits.

Third you can already use DEBUG_RESOLVER_TREE=1 while doing bundle install, which will give you all the information you have asked for.

I think this more of a documentation problem then a code one, I am happy to write docs which would explain how Bundler dependency works in general and debug commands they can use to verify their problems. I think this would be better because it will make people aware of how Bundler actually works and it would be better from both contribution and user point of view.

Happy to hear your thoughts.

[rosenfeld]

Ok, I believe it's okay not to display all versions, but maybe something like "tried 4.0.4 down to 3.0.0" would be helpful already. The main problem is that currently the message points activesupport =3.0.0 as a dependency of activemodel >= 3.0.0, which is not true.

When conflict happens it would help a lot to help the user to track down what exactly is conflicting since it's not always obvious. In that sense, mentioning DEBUG_RESOLVER_TREE or even pointing to a Bundler's conflict resolution page would be quite helpful.

I tried the DEBUG_RESOLVER_TREE option but I think it's too verbose and not exactly what I was looking for (of course, it's useful for debugging purposes, but not that much for conflict resolution in a higher level).

It would be awesome if the message could be written like:

activemodel (>= 3.0) ruby depends on
        activesupport (tried final releases from 4.0.4 down to 3.0.0) ruby

But if it's not possible, please don't tell that it depends on activesupport =3.0.0 because this is not true and very misleading as one might be looking at which dependency has this strong requirement on activesupport...