Refactor: Bundler load order and already activated errors

[ph]

I know that some of you have been plagued with already activated error for some time (@jordansissel)

 +Gem::LoadError: You have already activated stud 0.0.23, but your Gemfile requires stud 0.0.24. Prepending `bundle exec` to your command may solve this.
       +  block in setup at /Users/ph/es/logstash/build/logstash-7.0.0-alpha1-SNAPSHOT/vendor/bundle/jruby/2.3.0/gems/bundler-1.9.10/lib/bundler/runtime.rb:34
       +            each at org/jruby/RubyArray.java:1734
       +            each at /Users/ph/es/logstash/build/logstash-7.0.0-alpha1-SNAPSHOT/vendor/jruby/lib/ruby/stdlib/forwardable.rb:189
       +           setup at /Users/ph/es/logstash/build/logstash-7.0.0-alpha1-SNAPSHOT/vendor/bundle/jruby/2.3.0/gems/bundler-1.9.10/lib/bundler/run

I have a pretty good understanding of what is going wrong here; it's related to bundler/RubyGems and your default environment. When no bundler is configured for the current scripts, it will load RubyGems, by default RubyGems will add your local gems path as source and will read the specifications. This can cause already activated errors. This means our Logstash isn't correctly isolated from our local env; this often happens in rake tasks because we are not consistently calling bundler on them.

When working on https://github.com/elastic/logstash/issues/8170, I revisited our class and how we load stuff. I believe it's and would not require a lot of big changes.

What needs to be done:

• [ ] Remove the custom name for Gemfile.jruby-2.3.lock
• [ ] Centralize our require of files into a single point, an init or bootstrap file.
• [ ] Revisit to make sure we always activate the bundler environment even if we install

Harder or higher risk:

• [ ] Refactor our install all plugins rake task
• [ ] refactor the install and update to not use bundler's cli class, but either use the resolver or the same logic that we use for x-pack.

[ph]

I am adding more context, I ran into the already activated errors.

You have already activated stud 0.0.23, but your Gemfile requires stud 0.0.24

I've looked at the gemfile and the lock and we only mention the 0.0.24 gems nothing about 0.0.23. But in the vendor directory we have both versions installed.

 ph@sashimi  ~/es/logstash/build/logstash-7.0.0-alpha1-SNAPSHOT/vendor/bundle   fix/mark-gems-unlockable ± find .| grep stud
./jruby/2.3.0/cache/stud-0.0.24.gem
./jruby/2.3.0/gems/stud-0.0.23
./jruby/2.3.0/gems/stud-0.0.23/CHANGELIST
./jruby/2.3.0/gems/stud-0.0.23/lib
./jruby/2.3.0/gems/stud-0.0.23/lib/stud
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/buffer.rb
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/pool.rb
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/secret.rb
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/temporary.rb
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/trap.rb
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/try.rb
./jruby/2.3.0/gems/stud-0.0.23/lib/stud/with.rb
./jruby/2.3.0/gems/stud-0.0.23/LICENSE
./jruby/2.3.0/gems/stud-0.0.23/README.md
./jruby/2.3.0/gems/stud-0.0.24
./jruby/2.3.0/gems/stud-0.0.24/stud.gemspec
./jruby/2.3.0/specifications/stud-0.0.23.gemspec
./jruby/2.3.0/specifications/stud-0.0.24.gemspec

So something auto require the gemspec in the specifications folder before bundler is activated. Deleting the stud-0.0.23.gemspec fixes the already activated errors.

If bundler was correctly configured, it wouldn't care if multiples versions are present it will only take the one defined in the lockfile and ignore the other.

[ph]

Usually bundler will wipe any specifications present before trying to load the one defined in the gemfile. I wonder if its not because one of the file of this gem is already required in memory preventing from cleaning or reseting the specification. Just a theory.