Running appraisal specs with Bundler >= 2.4.0 broken

tisba commented 4 months ago

Hey there 👋 I'm trying to send a PR (https://github.com/thoughtbot/appraisal/pull/219) and I'd like to add specs. But I'm unable to run the current specs. What am I missing?

I did this:

git clone git@github.com:thoughtbot/appraisal.git
bundle

rake fails with various other warnings but also the same error (no surprise as the Rakefile does not seem to do something special).

$ rspec ./spec/acceptance/appraisals_file_bundler_dsl_compatibility_spec.rb:4 fails like this:

Run options: include {:locations=>{"./spec/acceptance/appraisals_file_bundler_dsl_compatibility_spec.rb"=>[4]}}
/Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/git/git_proxy.rb:354:in `allowed_with_path': The git source ../gems/omelette is not yet checked out. Please run `bundle install` before trying to start your application (Bundler::GitError)
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/git/git_proxy.rb:78:in `revision'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/git.rb:242:in `revision'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/git.rb:116:in `install_path'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/path.rb:134:in `expanded_path'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/path.rb:175:in `load_spec_files'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/git.rb:219:in `load_spec_files'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/path.rb:108:in `local_specs'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/git.rb:185:in `specs'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/rubygems_aggregate.rb:35:in `block (2 levels) in build_index'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/rubygems_aggregate.rb:33:in `each'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/rubygems_aggregate.rb:33:in `block in build_index'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/index.rb:9:in `build'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/rubygems_aggregate.rb:30:in `build_index'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/source/rubygems_aggregate.rb:12:in `initialize'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:935:in `new'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:935:in `source_requirements'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:543:in `resolution_packages'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:525:in `resolver'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:600:in `start_resolution'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:311:in `resolve'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:553:in `materialize'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:203:in `specs'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/definition.rb:270:in `specs_for'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/runtime.rb:18:in `setup'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler.rb:162:in `setup'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/setup.rb:26:in `block in <top (required)>'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/ui/shell.rb:159:in `with_level'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/ui/shell.rb:111:in `silence'
        from /Users/basti/.gem/ruby/3.3.0/gems/bundler-2.5.4/lib/bundler/setup.rb:26:in `<top (required)>'
        from <internal:/Users/basti/.rubies/ruby-3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:127:in `require'
        from <internal:/Users/basti/.rubies/ruby-3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:127:in `require'
        from /Users/basti/.rubies/ruby-3.3.0/lib/ruby/site_ruby/3.3.0/rubygems.rb:1375:in `<top (required)>'
        from <internal:gem_prelude>:2:in `require'
        from <internal:gem_prelude>:2:in `<internal:gem_prelude>'
F

Failures:

  1) Appraisals file Bundler DSL compatibility supports all Bundler DSL in Appraisals file
     Failure/Error:
       raise RuntimeError, <<-error_message.strip_heredoc
         Command #{command.inspect} exited with status #{exitstatus}. Output:
         #{output.gsub(/^/, '  ')}
       error_message

     RuntimeError:
       Command "bundle install --local" exited with status 1. Output:

     # ./spec/support/acceptance_test_helpers.rb:163:in `block (2 levels) in run'
     # ./spec/support/acceptance_test_helpers.rb:155:in `block in run'
     # ./spec/support/acceptance_test_helpers.rb:150:in `chdir'
     # ./spec/support/acceptance_test_helpers.rb:150:in `in_test_directory'
     # ./spec/support/acceptance_test_helpers.rb:154:in `run'
     # ./spec/acceptance/appraisals_file_bundler_dsl_compatibility_spec.rb:100:in `block (2 levels) in <top (required)>'

Finished in 7.35 seconds (files took 0.22456 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/acceptance/appraisals_file_bundler_dsl_compatibility_spec.rb:4 # Appraisals file Bundler DSL compatibility supports all Bundler DSL in Appraisals file

nickcharlton commented 4 months ago

I think it's a bundler incompatibility that you're seeing. I haven't been able to pin down exactly what it is, but I was working on this last week.

I can run bundle exec rspec on Ruby 3.3.0 and with:

$ bundle exec bundler --version
Bundler version 2.3.7

…and the tests pass, but different versions cause problems.

In addition, last week I was playing around with a branch I'd worked on ages ago for testing out GitHub Actions and seen the same.

Could you try replicating with one of the combinations (see the GHA branch), and see if that helps you?

tisba commented 4 months ago

That was a good hint, thanks. I was a little lost and couldn't really find a good starting point. I did some very corse search and found 2.3.27 to be working. I poked at some 2.4.x and 2.5.x versions, but was seeing the same error. I can try some more smart bisecting in the next days, if that helps. My guess is that there broken something between 2.3.27 (laest 2.3.x release) and 2.4.0.

Looking at https://github.com/rubygems/rubygems/blob/master/bundler/CHANGELOG.md#240-december-24-2022, I kind of have the suspicion that https://github.com/rubygems/rubygems/pull/4475 might be the reason. Bundler::Source::Git has been changed quite a bit, where the error message is coming from.

🤔

tisba commented 4 months ago

Forgot to mention: I tested bundler versions with this: bundle _2.4.0_ update --bundler && bundle && rspec ./spec/acceptance/appraisals_file_bundler_dsl_compatibility_spec.rb. Since bundler also manages itself, debugging things like that became rather easy :)

nickcharlton commented 4 months ago

Ah, hah, nice! It'd be great if you could spend the time tracking down what might have changed.

Whilst you're doing that, I'd be keep to hear if you have any ideas of what we could do to make these things a bit easier to debug, too.

tisba commented 4 months ago

This is what I have so far.

Minimal acceptance test:

require 'spec_helper'

describe 'Debugging Bundler Issues' do
  it 'does something' do
    build_git_gems %w(omelette)

    build_gemfile <<-Gemfile
      source 'https://rubygems.org'
      gem "omelette", git: "../gems/omelette"
    Gemfile

    run 'bundle install --local --verbose'
  end
end

Trigger the error via bundle _2.5.6_ exec rspec ./spec/acceptance/bundler_debug_spec.rb.

When you then go into tmp/stage and run bundle install --local --verbose you'll get the same error, but also see stderr from bundler:

Retrying `git clone --bare --no-hardlinks --quiet --no-tags --depth 1 --single-branch -- file://../gems/omelette /Users/basti/.gem/ruby/3.3.0/cache/bundler/git/omelette-0232b7510a79059d96d8bcbfb4a82ad436173b63` due to error (2/4): Bundler::Source::Git::GitCommandError Git error: command `git clone --bare --no-hardlinks --quiet --no-tags --depth 1 --single-branch -- file://../gems/omelette /Users/basti/.gem/ruby/3.3.0/cache/bundler/git/omelette-0232b7510a79059d96d8bcbfb4a82ad436173b63` in directory /Users/basti/.gem/ruby/3.3.0/cache/bundler/git/omelette-0232b7510a79059d96d8bcbfb4a82ad436173b63 has failed.
fatal: '/gems/omelette' does not appear to be a git repository
fatal: Could not read from remote repository.
[…]

BINGO!

Here is what happens:

Bundler is doing git clone --bare --no-hardlinks --quiet --no-tags --depth 1 --single-branch -- file://../gems/omelette […], to clone the git repo of the gem to the bundler cache using a shallow clone.
BUT the file:// scheme does not permit relative paths (see RFC8089 "The path component represents the absolute path to the file in the file system." and RFC1738).
The underlying problem is hidden because AcceptanceTestHelpers#run is unfortunately using Kernel#` which only captures standard out. In case of errors when invoking bundler, this is quite useless.

What does that mean?

tl;dr: I think Bundler does not support local git repositories to be defined via Gemfile (or it just broken). The docs are pretty clear though: For gems out of local git repositories you are supposed to do bundle config set local.GEM_NAME /path/to/local/git/repository.

I'll try to dig through the Bundler changelog and PRs to see if this was intentional or not. I suspect that this was an unintended side-effect introduced by https://github.com/rubygems/rubygems/pull/4475. As it was never documented/recommended to use gems from local git repositories in a different way.

IMO, btw: It should either work, or Bundler should raise an error, that local git repos are not supported, instead of trying to feed an invalid file:// URI to git.

nickcharlton commented 4 months ago

Oh wow! Great work. Thanks for digging into it.

thoughtbot / appraisal

Running appraisal specs with Bundler >= 2.4.0 broken #218

BINGO!

What does that mean?