Open aprescott opened 1 month ago
Since 2.2.4 includes #48 as the only documented change, I assume something there broke.
Meant to include other version information, apologies! 😅
$ bundle exec rspec --version
RSpec 3.13
- rspec-core 3.13.0
- rspec-expectations 3.13.1
- rspec-mocks 3.13.1
- rspec-rails 6.1.3
- rspec-support 3.13.1
$ ruby -v
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23]
Happy to provide any other extra information if it's helpful.
Please add the --seed NUMBER
parameter to the turbo_tests command. Before #48, seed was randomized by default, which was confusing. (ParallelTests and RSpec apply seed
only when option is used explicitly.)
Here's the output with the --seed
parameter
$ bundle exec turbo_tests --seed $RANDOM
Using recorded test runtime
3 processes for 3 specs, ~ 1 specs per process
Randomized with seed 5366
..........
Finished in 4.56 seconds (files took 3.12 seconds to load)
10 examples, 0 failures
Randomized with seed 5366
Output without the seed parameter
$ bundle exec turbo_tests
Using recorded test runtime
3 processes for 3 specs, ~ 1 specs per process
..........
Finished in 4.74 seconds (files took 3.11 seconds to load)
10 examples, 0 failures
Thanks for the bug report @aprescott and sorry for the inconvenience.
Please let me know if adding the --seed NUMBER
parameter works for you.
RSpec apply
seed
only when option is used explicitly.
This option is enabled for rspec
runs through the standard spec_helper.rb
config boilerplate:
config.order = :random
Kernel.srand config.seed
Moreover, turbo_tests
is using this config: the ordering of tests is still random when using turbo_tests
, it's just that now there's no seed value shown. If I use version 2.2.4 and place this in a file:
specify { puts 1; expect(1).to eq(1) }
specify { puts 2; expect(1).to eq(1) }
Then 1
and 2
will be printed in a random order for each run of turbo_tests
(without --seed
given), yet no seed is shown in 2.2.4.
If turbo_tests
doesn't display the random seed and requires that one be specified explicitly, then things are left in a surprising (to me) situation. With a test suite configured to use a random order:
rspec
requires no --seed
value, and will print the random seed used for a test run.turbo_tests
(2.2.4) will still run in a random order, but it won't show the seed used unless --seed
is given.That feels incorrect to me? I understand there are other bugs being addressed by the change in behavior here, but it effectively means that --seed
must be given solely to make the seed value display: if --seed
isn't given as a param, it is still random but the default behavior provides no automatic, direct way of reproducing an ordering that was used in the event of an order-dependent test failure.
As a user I think this makes turbo_tests
harder to use as a drop-in for rspec
: a dev cannot simply use turbo_tests
over rspec
, they must now ensure that --seed $RANDOM
or an equivalent is specified in both automated tooling (e.g. CI) and in all their own command runs, ensure it's part of their aliases, etc., or do something custom to print the seed value at the start/end of the tests.
Could turbo_tests
read the RSpec config, and use that to determine whether to act as if a --seed
has been given?
Could
turbo_tests
read the RSpec config, and use that to determine whether to act as if a--seed
has been given?
To expand on that a bit: basically if config.order
is :random
then act as if turbo_tests
was given --seed $RANDOM
(or similar, I'm not sure what RSpec itself does in Ruby for example). That would allow turbo_tests
to presumably fix the seed for each parallel run, printing the value before and after, preserving the expected behavior.
Version 2.2.4 appears to no longer display any seed information output, for success or failure, either at the beginning or the end of the run:
Compare that to 2.2.3, which outputs at the start and the end of the test, as expected: