OneOf encoding/decoding test failing on `Ruby 3.4.0dev`?

[maximecb]

I just did a git pull and found that there was a stray binding.irb in message_test.rb: https://github.com/Shopify/protoboeuf/commit/ca3e5d415b1a92ec2ca69d0be8ce7779d1a3f1a0

Upon removing that, I found that the tests was a failing test, but the error messages are still getting swallowed by rake test, which makes debugging unnecessarily annoying. I've commented out the minitest/color import for now, and I think we should leave it disabled for now: https://github.com/Shopify/protoboeuf/commit/b0fc02c850b320317a44153f8f5f9bafc6bb15e6

There's currently a failure in the OneOf encoder/decoder test that shows up with the latest Ruby 3.4.0dev (YJIT disabled). I did a rake clobber and also tried nuking my local clone from orbit. This mysterious failure doesn't show up on the CI, so it seems like it might be something specific to Ruby master? (argh!) I might try installing 3.3.0 after lunch to see if the problem persists, if that's useful 🤔

  1) Failure:
MessageTest#test_encode_oneof [test/message_test.rb:658]:
--- expected
+++ actual
@@ -1 +1 @@
-{:id=>"abcd", :shop_id=>5678, :oneof_u32=>0, :oneof_msg=>nil, :oneof_str=>"hello world", :boolean=>true}
+{:id=>"abcd", :shop_id=>5678, :oneof_str=>"hello world", :boolean=>true}

So @paracycle maybe you found/triggered a bug in the latest Ruby master?

[maximecb]

I can confirm that the test failure happens with Ruby master but not with Ruby-3.3.0. Testing on macOS, no YJIT.

[paracycle]

Which Ruby 3.4 version are you running btw? What is ruby -v for you?

[maximecb]

It's Ruby master 0107954f257af6fd4cb280aa36b9b320795c0a86 ruby 3.4.0dev (2024-04-09T16:29:01Z master 0107954f25) [arm64-darwin23]

Built on macOS. Doesn't seem to matter if it's built in dev mode or not.

[paracycle]

I can't recreate that problem:

$ chruby ruby-head

ufuk in protoboeuf on  main with [v3.4.0dev]                                                                      
$ bundle
Bundler 2.6.0.dev is running, but your lockfile was generated with 2.5.7. Installing Bundler 2.5.7 and restarting using that version.
Fetching gem metadata from https://rubygems.org/.
Fetching bundler 2.5.7
Installing bundler 2.5.7
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Fetching rake 13.2.0
Installing rake 13.2.0
Fetching benchmark-ips 2.13.0
Fetching google-protobuf 4.26.1
Fetching prettier_print 1.2.1
Fetching minitest-color 0.0.2
Installing benchmark-ips 2.13.0
Installing google-protobuf 4.26.1 with native extensions
Installing prettier_print 1.2.1
Fetching syntax_tree 6.2.0
Installing minitest-color 0.0.2
Installing syntax_tree 6.2.0
Bundle complete! 6 Gemfile dependencies, 8 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

ufuk in protoboeuf on  main with [v3.4.0dev]
$ bundle exec rake
mkdir -p lib/proto
protoc test/fixtures/test.proto --ruby_out=lib/proto
/Users/ufuk/.rubies/ruby-head/bin/ruby -w -I"lib:test" /Users/ufuk/.gem/ruby/ruby-head/gems/rake-13.2.0/lib/rake/rake_test_loader.rb "test/codegen_test.rb" "test/message_test.rb" "test/parser_test.rb" "test/visitor_test.rb" "test/well_known_types_test.rb"
Run options: --seed 59656

# Running:

.................................................................................................................

Finished in 1.576390s, 71.6828 runs/s, 204.2642 assertions/s.

113 runs, 322 assertions, 0 failures, 0 errors, 0 skips

ufuk in protoboeuf on  main with [v3.4.0dev]
$ ruby -v
ruby 3.4.0dev (2024-04-09T16:29:01Z master 0107954f25) [arm64-darwin23]

[paracycle]

Are you sure you are running rake test with bundle exec btw?

[maximecb]

The test does pass with bundle exec rake test. What's the difference? Does it change the gem path? Why does rake test also run without bundle exec?

[paracycle]

The test does pass with bundle exec rake test. What's the difference? Does it change the gem path? Why does rake test also run without bundle exec?

Ah, yeah that's quite nuanced. rake test finds the rake executable in your gem executable paths and runs that. That could be any arbitrary version of Rake that is installed in your local gem collection, but is usually the binary that is exported from latest version of rake that is installed on your system.

However, bundle exec rake test does the bundle setup routine, which limits the load paths to only the gem paths of the exact versions of the gems that are a part of your Gemfile.lock. So, it provides determinism and isolation from side-effects.

If typing bundle exec is tiring, a common thing to do is to generate a binstub for rake using bundle binstub rake which will create a bin/rake script which does the bundler setup automatically. You will then have to use bin/rake test to run your tests.

TL;DR: It is very important to always use bundle exec when running commands inside a project with a Gemfile to make sure that you operate within the isolation confines of the set of gems listed in the gem file.