search

KnapsackPro / knapsack_pro-ruby

Knapsack Pro gem splits tests across parallel CI nodes and makes sure that tests run in optimal time
https://knapsackpro.com
MIT License
132 stars 27 forks source link

Split by example => uninitialized constant KnapsackPro::BaseAllocatorBuilder::RSpec #119

Closed dannyfallon closed 4 years ago

dannyfallon commented 4 years ago

When using the following with Knapsack Pro 2.1.0 I receive an uninitialized constant exception:

KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true DISABLE_SPRING=true bundle exec rake knapsack_pro:rspec

Output/Trace: 

+ bundle exec rake knapsack_pro:rspec
--
  | .......rake aborted!
  | NameError: uninitialized constant KnapsackPro::BaseAllocatorBuilder::RSpec
  | /bundler_cache/ruby/2.6.0/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/active_support.rb:79:in `block in load_missing_constant'
  | /bundler_cache/ruby/2.6.0/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/active_support.rb:8:in `without_bootsnap_cache'
  | /bundler_cache/ruby/2.6.0/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/active_support.rb:79:in `rescue in load_missing_constant'
  | /bundler_cache/ruby/2.6.0/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/active_support.rb:58:in `load_missing_constant'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/base_allocator_builder.rb:37:in `fast_and_slow_test_files_to_run'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/allocator_builder.rb:5:in `allocator'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/base_runner.rb:10:in `initialize'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/rspec_runner.rb:8:in `new'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/rspec_runner.rb:8:in `run'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/tasks/rspec.rake:5:in `block (2 levels) in <main>'
  | /bundler_cache/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'

I believe the problem persists in Queue mode, though with a different exception:


I, [2020-07-17T17:19:30.163122 #830]  INFO -- : [knapsack_pro] Generating RSpec test examples JSON report for slow test files to prepare it to be split by test examples (by individual 'it's. Thanks to that a single slow test file can be split across parallel CI nodes). Analyzing 33 slow test files.
--
  | ..........rake aborted!
  | There was problem to generate test examples for test suite
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/test_case_detectors/rspec_test_example_detector.rb:36:in `generate_json_report'

<...snipped logs...>

Tasks: TOP => knapsack_pro:rspec_test_example_detector
--
  | (See full trace by running task with --trace)
  | rake aborted!
  | Could not generate JSON report for RSpec. Rake task failed when running RACK_ENV=test RAILS_ENV=test bundle exec rake knapsack_pro:rspec_test_example_detector
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/base_allocator_builder.rb:53:in `fast_and_slow_test_files_to_run'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/queue_allocator_builder.rb:5:in `allocator'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/queue/base_runner.rb:15:in `initialize'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/queue/rspec_runner.rb:14:in `new'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/queue/rspec_runner.rb:14:in `run'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/tasks/queue/rspec.rake:6:in `block (3 levels) in <main>'
  | /bundler_cache/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
  | /usr/local/bundle/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
  | /usr/local/bundle/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
  | /usr/local/bundle/bin/bundle:23:in `load'
  | /usr/local/bundle/bin/bundle:23:in `<main>'
  | Tasks: TOP => knapsack_pro:queue:rspec
  | (See full trace by running task with --trace)

The removal of the KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES fixes the problem.

ArturT commented 4 years ago

Hi @dannyfallon can you double-check knapsack_pro gem is in test and development group in Gemfile?

Can you ensure you set RAILS_ENV=test on CI. It's must-have for Queue Mode so the process starts running in test environment instead of development.

This should help with bug NameError: uninitialized constant KnapsackPro::BaseAllocatorBuilder::RSpec.

You can also check this: https://knapsackpro.com/faq/question/how-to-split-slow-rspec-test-files-by-test-examples-by-individual-it#why-i-see-error-dont-know-how-to-build-task-knapsack_pro-rspec_test_example_detector and set KNAPSACK_PRO_RSPEC_TEST_EXAMPLE_DETECTOR_PREFIX="" so then the failing rake task RACK_ENV=test RAILS_ENV=test bundle exec rake knapsack_pro:rspec_test_example_detector will run without prefix bundle exec. Today another user reported that this helped him run feature RSpec split by test examples.

Please let me know if this will help.

dannyfallon commented 4 years ago

Hey @ArturT, thanks for coming back to me.

I can confirm we've got the following in our Gemfile:

group :development, :test do
  gem 'knapsack_pro'
end

Queue and Regular mode work fine as long as we don't pass in KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES. This fails in both Buildkite and locally (I am passing in branch/commit info as fake args).

KNAPSACK_PRO_BRANCH=df/test KNAPSACK_PRO_COMMIT_HASH=deadbeef KNAPSACK_PRO_CI_NODE_TOTAL=2 KNAPSACK_PRO_CI_NODE_INDEX=0 KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true bundle exec rake knapsack_pro:rspec
rake aborted!
NameError: uninitialized constant KnapsackPro::BaseAllocatorBuilder::RSpec
/Users/danny/.rbenv/versions/2.6.5/bin/bundle:23:in `load'
/Users/danny/.rbenv/versions/2.6.5/bin/bundle:23:in `<main>'

Caused by:
NameError: uninitialized constant KnapsackPro::BaseAllocatorBuilder::RSpec
/Users/danny/.rbenv/versions/2.6.5/bin/bundle:23:in `load'
/Users/danny/.rbenv/versions/2.6.5/bin/bundle:23:in `<main>'
Tasks: TOP => knapsack_pro:rspec
(See full trace by running task with --trace)
ArturT commented 4 years ago

I'm wondering, maybe the issue is RSpec because there is no class like KnapsackPro::BaseAllocatorBuilder::RSpec.

https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/base_allocator_builder.rb#L37

I guess below line might be problematic 

unless Gem::Version.new(RSpec::Core::Version::STRING) >= Gem::Version.new('3.3.0')

because your previous backtrace indicate that 

  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/base_allocator_builder.rb:37:in `fast_and_slow_test_files_to_run'

RSpec is not loaded so Ruby tried to find it inside of KnapsackPro::BaseAllocatorBuilder.

Could you try export ENV vars so this may we would ensure they are also passed down to process rake knapsack_pro:rspec_test_example_detector that is started from inside of knapsack_pro Queue Mode rake task.

export RAILS_ENV=test
export KNAPSACK_PRO_BRANCH=something
...

Also, I'm curious how did you add rspec gem in Gemfile. Is it in test group only or in test & development? Just in test group should be enough if we export RAILS_ENV=test.

dannyfallon commented 4 years ago

This was because RAILS_ENV=test wasn't exported and wasn't picked up. Sorry for the false bug report Artur 👍

ArturT commented 4 years ago

@dannyfallon ok. I'm glad it works now. If you need more help with setup just contact me at support@knapsackpro.com :)

dhinus commented 3 years ago

I'm having a very similar problem, only the exception is

NameError: uninitialized constant RSpec

RAILS_ENV=test is exported and gem 'knapsack_pro' is included in the test section of the Gemfile.

If I remove KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true then everything works fine.

The stacktrace is pointing to knapsack_pro/base_allocator_builder.rb:37 as in the discussion above:

NameError: uninitialized constant RSpec
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/active_support.rb:61:in `block in load_missing_constant'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/active_support.rb:17:in `allow_bootsnap_retry'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.4.6/lib/bootsnap/load_path_cache/core_ext/active_support.rb:60:in `load_missing_constant'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/knapsack_pro-2.8.0/lib/knapsack_pro/base_allocator_builder.rb:37:in `fast_and_slow_test_files_to_run'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/knapsack_pro-2.8.0/lib/knapsack_pro/allocator_builder.rb:5:in `allocator'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/knapsack_pro-2.8.0/lib/knapsack_pro/runners/base_runner.rb:10:in `initialize'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/knapsack_pro-2.8.0/lib/knapsack_pro/runners/rspec_runner.rb:8:in `new'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/knapsack_pro-2.8.0/lib/knapsack_pro/runners/rspec_runner.rb:8:in `run'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/knapsack_pro-2.8.0/lib/tasks/rspec.rake:5:in `block (2 levels) in <main>'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/home/travis/build/loco2/loco2/vendor/bundle/ruby/2.7.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
ArturT commented 3 years ago

Hi @dhinus

If you already have RAILS_ENV=test then you maybe forgot to add knapsack_pro gem to the development group as well:

Gemfile
group :test, :development do
  gem 'knapsack_pro'
end

Or you need to add rspec gem in test and development group too.

NameError: uninitialized constant RSpec may indicate that RSpec is not loaded.

dhinus commented 3 years ago

@ArturT Thanks for replying. The solution was just before my eyes: the RSpec gem was in the right group in the Gemfile, but with require: false:

group :test do
  gem 'rspec',                             '~> 3.9', require: false

This wasn't an issue until now because we have require 'rspec' in spec/spec_helper.rb. It looks like knapsack_pro/base_allocator_builder.rb is called before that.

Simply removing require: false fixed the issue!

ArturT commented 3 years ago

Awesome @dhinus Thanks for sharing your solution to this problem. It might help someone in the future ;)