bootsnap/setup cannot load msgpack

fxn commented 4 months ago

Disclaimer

I discovered this messing around, but do not personally need it fixed in practice.

Reporting just in case it is of the interest of the project.

I made sure the issue is in bootsnap

I did not investigate if the issue is in Bootsnap itself or perhaps in RubyGems.

Steps to reproduce

Minimal example, does not need a Gemfile, can be reproduced in an empty directory:

ruby <<EOS
  require 'tmpdir'

  Dir.mktmpdir do |cache|
    ENV['BOOTSNAP_CACHE_DIR'] = cache

    require 'bootsnap/setup'
  end
EOS

Expected behavior

We get a prompt back.

Actual behavior

While the gem msgpack is installed:

% ruby -rmsgpack -e 'p MessagePack::VERSION'
"1.7.2"

Bootsnap initialization is not able to load it:

<internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require': cannot load such file -- msgpack (LoadError)
    from <internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
    from <internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/compile_cache/yaml.rb:59:in `init!'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/compile_cache/yaml.rb:38:in `install!'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/compile_cache.rb:25:in `setup'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap.rb:68:in `setup'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap.rb:112:in `default_setup'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/setup.rb:5:in `<top (required)>'
    from <internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
    from <internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
    from -:7:in `block in <main>'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/3.3.0/tmpdir.rb:99:in `mktmpdir'
    from -:3:in `<main>'
<internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- msgpack (LoadError)
    from <internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:17:in `require'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/compile_cache/yaml.rb:59:in `init!'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/compile_cache/yaml.rb:38:in `install!'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/compile_cache.rb:25:in `setup'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap.rb:68:in `setup'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap.rb:112:in `default_setup'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/setup.rb:5:in `<top (required)>'
    from <internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
    from <internal:/Users/fxn/.rbenv/versions/3.3.0/lib/ruby/site_ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
    from -:7:in `block in <main>'
    from /Users/fxn/.rbenv/versions/3.3.0/lib/ruby/3.3.0/tmpdir.rb:99:in `mktmpdir'
    from -:3:in `<main>'

System configuration

Bootsnap version:

1.18.3

Ruby version:

3.3.0

RubyGems version:

3.5.15

casperisfine commented 4 months ago

Alrigth, I tracked it down to Bootsnap::ExplicitRequire.with_gems: https://github.com/Shopify/bootsnap/commit/07bd09e06913ac172db0ba0d2a0c555f51a451d2

When calling require "msgpack" the Rubygems require patch essentially calls Gem.try_activate("msgpack") that adds message pack's directories in the $LOAD_PATH, so it can be required.

Then ExplicitRequire.with_gems does a hack to try to speedup requiring depedencies. It saves the $LOAD_PATH, then reduce it, require and restore the $LOAD_PATH.

But this hack wasn't tested in bundler-less environment, so it never expected the call to require to add entries in the $LOAD_PATH, so when it restores it, the paths added by rubygems are lost and it causes the bug you showcase.

I can fix it by explicitly activating the gems passed to with_gems before doing the $LOAD_PATH trick.

fxn commented 4 months ago

:heart:

Shopify / bootsnap