Newrelic 9.10.0 breaks our CI - caused by a collision with the 'bootstrap' gem

[dorner]

Description

On a dependabot PR for the new version of the agent, all GitHub Action CI steps fail with the following error:

New Relic entrypoint at /home/runner/work/human-essentials/human-essentials/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.10.0/lib/bootstrap.rb encountered an issue:
    /home/runner/work/human-essentials/human-essentials/vendor/bundle/ruby/3.2.0/gems/newrelic_rpm-9.10.0/lib/bootstrap.rb is meant to be required via the RUBYOPT env var

No previous versions caused these errors.

Sample run: https://github.com/rubyforgood/human-essentials/actions/runs/9297054369/job/25667726353

Expected Behavior

No crashes!

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5): Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration[bot]]

https://new-relic.atlassian.net/browse/NR-276398

[fallwith]

Hi @dorner,

Sorry to hear that the new agent version is causing this problem. That bootstrap.rb file is new in v9.10.0 and is meant only to be used with some experimental new use cases. And conversely it is designed to be invisible to all existing use cases (shouldn't be read or required by anything) and in fact will complain as you've seen if it was loaded in an unexpected way.

We didn't expect your use case to load bootstrap.rb, but somehow it was loaded.

From looking at your failed run, it looks like that job only does 2 things:

1. bundle exec rake db:create
2. bundle exec rake db:schema:load

Using a Rails v7.1 based app, I bundled the New Relic Ruby agent v9.10 and performed those commands and didn't see the same problem. I tried both with a non-existent database that needed to be created and against an existing database.

Is it possible that your app is engaging some other code when those rake tasks are performed? Are you aware of anything that might be globbing up all .rb files and calling load or require on them?

We will keep investigating regardless, but if you could provide us with some reproduction steps that would be helpful. Perhaps the steps could start with rails new and go from there.

If we need to, we can rework bootstrap.rb to not perform exit, but we'd like to first better understand how you reached that line.

[dorner]

Is it possible that there is some conflict with the bootstrap gem? We have that in our Gemfile. I'm wondering if a require "bootstrap" is somehow getting confused and grabbing your file.

I don't see any instance of load or require that seems to be working on individual files inside gems.

Btw this is an open source / non-profit application, so you are free to clone it yourself! https://github.com/rubyforgood/human-essentials/

[fallwith]

Ah, thanks, @dorner. Having bootstrap listed in Gemfile does indeed seem necessary to reproduce the problem. We can repro now.

My hypothesis: the New Relic gem's lib/ dir and the bootstrap gem's lib/ dir are both in the Ruby lib path and when require 'bootstrap' is performed, Ruby is finding our lib/bootstrap.rb file first and hitting our file's exit.

We'll work on a fix that could involve renaming bootstrap.rb, relocating it to anywhere but the root of lib, silently returning instead of calling warn and exit, or a combination of these.

For now, I consider v9.10 to have broken compatibility with the bootstrap gem. Users of the bootstrap gem will need to pin the New Relic Ruby agent v9.9 until we can publish a fix.

Thanks very much for reporting this and for your patience with us. And thanks so much for your Ruby for Good efforts.

[fallwith]

@dorner I have created PR #2676 to fix the issue. It will need to be reviewed and approved by the other project maintainers before it can make its way into a new release.

If you would like to test the fix prior to it making its way out into a published gem release, you can reference this repo's "bootstrap" branch like so:

# Gemfile

gem 'newrelic_rpm',
  github: 'newrelic/newrelic-ruby-agent',
  branch: 'bootstrap'

You are of course welcome to simply stay on agent v9.9.0 if you'd prefer.

Either way, we'll be sure to post an update here once the fix makes its way out to RubyGems.

Thanks again for the bug report!

[juno]

@fallwith Just FYI. My project faced with the same situation as dorner's, and it solved with your bootstrap branch. Thank you for quick fix!

Also, Thank you for reporting this, @dorner . 👏

[dorner]

Amazing, thanks so much for the prompt response and fix!

[fallwith]

The fix has been merged. We'll post another update here once a new release has been pushed to RubyGems.org.

[fallwith]

@dorner @juno @marlo-longley Version v9.10.1 of the New Relic Ruby agent has now been released to RubyGems.org. All bootstrap users should now able to upgrade past v9.9.0 by going with v9.10.1. Thanks again!

[dorner]

Awesome!

[fallwith]

cc @sidane, @aseroff, @thestelz v9.10.1 is now available with the fix