serpapi / turbo_tests

Run RSpec tests on multiple cores. Like parallel_tests but with incremental summarized output. Originally extracted from the Discourse and Rubygems source code.
https://rubygems.org/gems/turbo_tests
MIT License
173 stars 26 forks source link

Add option to print failing group #34

Open mRudzki opened 1 year ago

mRudzki commented 1 year ago

Why this option is useful? If you stumble across random failures you can debug it easier.

mRudzki commented 1 year ago

I did it separately from https://github.com/serpapi/turbo_tests/pull/33 but maybe I shouldn't . Many projects I worked on have some old/poorly written specs that fail only on specific occasion.

Example 1

it "one" do
  record = build :record
  expect { record.save }.to eq(true)
end

it "two" do
  record = build :record, id: 1
  expect { record.save }.to eq(true)
end

In this simple example, order "two, one" will succeed but "one, two" will fail.

Example 2

A more real world (I had that exact problem on some old spec last month) example is setting thread variables and forgetting to unset them later.

# file1.rb
let(:client) { create :client }
...
it "saves currrent_user_id in execution_context_store" do
  expect { complex_class.new(user: client).call }.to change(Thread.current[:execution_context_store]).from(nil).to({client: client})
end

# file2.rb
  after do
    Thread.current[:execution_context_store] = {}
  end

  describe ".store" do
    before do
      Thread.current[:execution_context_store] = { some: "value" }
    end

    it "returns value stored in current thread 'execution_context_store'" do
      expect(described_class.store).to eq(some: "value")
    end
  end

If file1.rb will be on same thread will be before file2.rb and won't be cleared with "after" we will have an error.

Why would it be beneficial?

Printing failing group, prints tests that were in that group. If we combine it with seed option, we can do bundle exec rspec --bisect and get the failing and specs it is depending on. Without it we have to run the whole suite, and even if we'll know failing seed on turbo_tests we won't be able to bisect on it. Here is why:

.
├── spec
│   ├── services
│   │   ├── file1.rb
│   │   ├── file2.rb
│   │   ├── file3.rb
│   │   ├── file4.rb
└── ...

Let's assume we have an example 2 but we also have:

# file3.rb
  after do
    Thread.current[:execution_context_store] = {}
  end
 ...

Let's assume turbo_tests seed 4321 -n 2 will return order: Thread1: "file1.rb, file2.rb", Thread2: "file3.rb, file4.rb" Result fail Let's assume "normal" rspec seed 4321 will return order: "file1.rb, file3.rb, file2.rb, file4.rb" Result: no fail because file3.rb prevented failure