dgollahon / rspectre

Exorcise the apparitions haunting your tests.
MIT License
75 stars 10 forks source link

Does not work properly with RSpec arguments --seed and --bisect #40

Open bquorning opened 4 years ago

bquorning commented 4 years ago

Given the following file spec/foo_spec.rb

class Foo
  class << self
    attr_accessor :bar
  end
  self.bar = 1
end

describe Foo do
  let(:unused) { "Hello" }

  it "is 1" do
    expect(Foo.bar).to eq(1)
  end

  it "is 1000" do
    Foo.bar = 1000
    expect(Foo.bar).to eq(1000)
  end
end

The following command will expose that Foo.bar = 1000 is changing global state: rspec --seed 4.

If I were to run rspectre on this test “suite” (expecting it to locate the line let(:unused) { "Hello" }), I would give a --rspec flag when calling the command. But I get various unexpected outcomes depending on which RSpec specific arguments I pass along:

❯ rspectre --rspec '--bisect --seed 4'
Running the specs failed. Either your tests do not pass normally or this is a bug in RSpectre.

RSpec Output:
---
Bisect started using options: "--seed 4"
Running suite to find failures...
Bisect failed! Failed to get results from the spec run. Spec run output:

No examples found.
❯ rspectre --rspec '--seed 4'
No unused test setup detected.
❯ rspectre --rspec '--bisect --seed 4 -- spec'
No unused test setup detected.
❯ rspectre --rspec '--seed 4 -- spec'
Running the specs failed. Either your tests do not pass normally or this is a bug in RSpectre.

RSpec Output:
---

Randomized with seed 4
.F

Failures:

  1) Foo is 1
     Failure/Error: expect(Foo.bar).to eq(1)

       expected: 1
            got: 1000

       (compared using ==)
     # ./spec/foo_spec.rb:12:in `block (2 levels) in <top (required)>'
     # /Users/bquorning/.gem/ruby/2.6.6/gems/rspectre-0.0.3/lib/rspectre/runner.rb:56:in `run_specs'
     # /Users/bquorning/.gem/ruby/2.6.6/gems/rspectre-0.0.3/lib/rspectre/runner.rb:15:in `lint'

Finished in 0.0131 seconds (files took 0.08895 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./spec/foo_spec.rb:11 # Foo is 1

Randomized with seed 4

In conclusion, it seems like --seed only works if you also pass a path-like argument. And --bisect does not work at all.