Seed value is no longer printed starting in version 2.2.4

[aprescott]

Version 2.2.4 appears to no longer display any seed information output, for success or failure, either at the beginning or the end of the run:

Finished in 7.38 seconds (files took 3.17 seconds to load)
467 examples, 0 failures

Finished in 5.24 seconds (files took 1.98 seconds to load)
467 examples, 1 failure

Failed examples:

rspec ./spec/models/foo_spec.rb:56 # 1 is expected to eq 2

Compare that to 2.2.3, which outputs at the start and the end of the test, as expected:

1 processes for 1 specs, ~ 1 specs per process

Randomized with seed 21152

# ...

Failures:

  1) Foo is expected to eq 2
     Failure/Error: specify { expect(1).to eq(2) }

       expected: 2
            got: 1

       (compared using ==)
     # ...

Finished in 5.47 seconds (files took 2 seconds to load)
467 examples, 1 failure

Failed examples:

rspec ./spec/models/foo_spec.rb:56 # Foo is expected to eq 2

Randomized with seed 21152

1 processes for 1 specs, ~ 1 specs per process

Randomized with seed 37227

# ...

Finished in 5.7 seconds (files took 2.06 seconds to load)
467 examples, 0 failures

Randomized with seed 37227

[aprescott]

Since 2.2.4 includes #48 as the only documented change, I assume something there broke.

[aprescott]

Meant to include other version information, apologies! 😅

$ bundle exec rspec --version
RSpec 3.13
  - rspec-core 3.13.0
  - rspec-expectations 3.13.1
  - rspec-mocks 3.13.1
  - rspec-rails 6.1.3
  - rspec-support 3.13.1

$ ruby -v
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23]

Happy to provide any other extra information if it's helpful.

[ilyazub]

Please add the --seed NUMBER parameter to the turbo_tests command. Before #48, seed was randomized by default, which was confusing. (ParallelTests and RSpec apply seed only when option is used explicitly.)

Here's the output with the --seed parameter

$ bundle exec turbo_tests --seed $RANDOM
Using recorded test runtime
3 processes for 3 specs, ~ 1 specs per process

Randomized with seed 5366
..........

Finished in 4.56 seconds (files took 3.12 seconds to load)
10 examples, 0 failures

Randomized with seed 5366

Output without the seed parameter

$ bundle exec turbo_tests 
Using recorded test runtime
3 processes for 3 specs, ~ 1 specs per process
..........

Finished in 4.74 seconds (files took 3.11 seconds to load)
10 examples, 0 failures

Thanks for the bug report @aprescott and sorry for the inconvenience.

Please let me know if adding the --seed NUMBER parameter works for you.

[aprescott]

RSpec apply seed only when option is used explicitly.

This option is enabled for rspec runs through the standard spec_helper.rb config boilerplate:

config.order = :random
Kernel.srand config.seed

Moreover, turbo_tests is using this config: the ordering of tests is still random when using turbo_tests, it's just that now there's no seed value shown. If I use version 2.2.4 and place this in a file:

specify { puts 1; expect(1).to eq(1) }
specify { puts 2; expect(1).to eq(1) }

Then 1 and 2 will be printed in a random order for each run of turbo_tests (without --seed given), yet no seed is shown in 2.2.4.

If turbo_tests doesn't display the random seed and requires that one be specified explicitly, then things are left in a surprising (to me) situation. With a test suite configured to use a random order:

• Running rspec requires no --seed value, and will print the random seed used for a test run.
• turbo_tests (2.2.4) will still run in a random order, but it won't show the seed used unless --seed is given.

That feels incorrect to me? I understand there are other bugs being addressed by the change in behavior here, but it effectively means that --seed must be given solely to make the seed value display: if --seed isn't given as a param, it is still random but the default behavior provides no automatic, direct way of reproducing an ordering that was used in the event of an order-dependent test failure.

As a user I think this makes turbo_tests harder to use as a drop-in for rspec: a dev cannot simply use turbo_tests over rspec, they must now ensure that --seed $RANDOM or an equivalent is specified in both automated tooling (e.g. CI) and in all their own command runs, ensure it's part of their aliases, etc., or do something custom to print the seed value at the start/end of the tests.

Could turbo_tests read the RSpec config, and use that to determine whether to act as if a --seed has been given?

[aprescott]

Could turbo_tests read the RSpec config, and use that to determine whether to act as if a --seed has been given?

To expand on that a bit: basically if config.order is :random then act as if turbo_tests was given --seed $RANDOM (or similar, I'm not sure what RSpec itself does in Ruby for example). That would allow turbo_tests to presumably fix the seed for each parallel run, printing the value before and after, preserving the expected behavior.