[Rails] FactoryGirl dynamic attributes cause neutral spec failures

[rdodson41]

In my Rails application, I use FactoryGirl to create test objects for my specs. It appears that FactoryGirl dynamic attributes, such as, for example, identifier { SecureRandom.uuid }, cause neutral spec failures when they are run via Mutant.

The following is the output of bundle exec mutant --use rspec OrdersController#index:

$ bundle exec mutant --use rspec OrdersController#index
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController#index]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
(00/21) 100% - killtime: 0.00s runtime: 0.00s overhead: 0.00s
(20/21)  95% - killtime: 3.45s runtime: 1.01s overhead: -2.44s
OrdersController#index:/Users/rdodson/Developer/marketplaces/app/controllers/orders_controller.rb:2
- rspec:18:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes should respond with orders
- rspec:19:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes should respond with orders
neutral:OrdersController#index:/Users/rdodson/Developer/marketplaces/app/controllers/orders_controller.rb:2:a8c5f
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :index,
  s(:args),
  s(:send, nil, :render,
    s(:hash,
      s(:pair,
        s(:sym, :json),
        s(:send,
          s(:const, nil, :Order), :paginate,
          s(:send, nil, :pagination_attributes))))))
Unparsed Source:
def index
  render(json: Order.paginate(pagination_attributes))
end
Test Result:
- 2 @ runtime: 0.169046
  - rspec:18:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes should respond with orders
  - rspec:19:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes should respond with orders
Test Output:
end of file reached
-----------------------
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController#index]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
Subjects:        1
Mutations:       21
Results:         21
Kills:           20
Alive:           1
Runtime:         1.01s
Killtime:        3.45s
Overhead:        -70.68%
Mutations/s:     20.76
Coverage:        95.24%
Coverage report generated for RSpec to /Users/rdodson/Developer/marketplaces/coverage. 151 / 433 LOC (34.87%) covered.

Test Output: end of file reached, indicative of an EOFError, seems like a bug to me. My FactoryGirl factories are as follows:

FactoryGirl.define do
  factory :marketplace do
    sequence :remote_id
    name { Faker::Company.name }
  end
end

FactoryGirl.define do
  factory :order do
    marketplace
    identifier { SecureRandom.uuid }
  end
end

I removed { and } from both the Marketplace and Order factories, turning their dynamic attributes into static attributes:

FactoryGirl.define do
  factory :marketplace do
    sequence :remote_id
    name Faker::Company.name
  end
end

FactoryGirl.define do
  factory :order do
    marketplace
    identifier SecureRandom.uuid
  end
end

As a result, bundle exec mutant --use rspec OrdersController#index passed:

$ bundle exec mutant --use rspec OrdersController#index
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController#index]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
(00/21) 100% - killtime: 0.00s runtime: 0.00s overhead: 0.00s
(09/21) 100% - killtime: 2.75s runtime: 1.01s overhead: -1.74s
(21/21) 100% - killtime: 5.10s runtime: 2.01s overhead: -3.09s
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController#index]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
Subjects:        1
Mutations:       21
Results:         21
Kills:           21
Alive:           0
Runtime:         2.01s
Killtime:        5.10s
Overhead:        -60.52%
Mutations/s:     10.43
Coverage:        100.00%
Coverage report generated for RSpec to /Users/rdodson/Developer/marketplaces/coverage. 151 / 433 LOC (34.87%) covered.

Has anyone else experienced this issue? Do we think that this is a bug in Mutant or upstream? I am using Ruby v2.3.3, Rails v5.1.3, FactoryGirl v4.8.0, and Mutant v0.8.14. As a side note, my entire RSpec suite passes with either of the above FactoryGirl configurations.

If anyone can provide some insight into this issue, then I would be happy to help fix it, but I am, at the moment, stuck with regard to tracking down its cause.

[mbj]

@rdodson41 I doubt this is a bug in mutant, and more a bug in your specs that fails mutant invariants.

I suspect that you leak instances generated with FB and this does not get detected in case they have static attributes, where with dynamic attributes under concurrency it causes mismatches between (leaky) spec setup and spec execution.

The error message in mutant should be updated to NOT point at unparser / mutant bugs anymore as all these had been tracked down and elimianted.

Instead neutral failures these days are almost always an spec environment invariant violation detected by mutant.

You can test if the concurrency invariant is violated via reverting to dynamic attributes, but use --jobs 1.

Other invariants are harder to test. The fact the rspec output is not properly captured could mean your test crashed hard (or still a mutant bug I'd not exclude entirely as possiblity).

In case you can provide a minimal example in a public repo I can track it down.

[rdodson41]

@mbj --jobs 1 has no effect:

$ bundle exec mutant --use rspec --jobs 1 OrdersController#index
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController#index]>
Integration:     Mutant::Integration::Rspec
Jobs:            1
Includes:        []
Requires:        []
(00/21) 100% - killtime: 0.00s runtime: 0.00s overhead: 0.00s
(07/21)  87% - killtime: 0.91s runtime: 1.01s overhead: 0.09s
(16/21)  94% - killtime: 1.93s runtime: 2.01s overhead: 0.08s
(20/21)  95% - killtime: 2.38s runtime: 3.01s overhead: 0.63s
OrdersController#index:/Users/rdodson/Developer/marketplaces/app/controllers/orders_controller.rb:2
- rspec:18:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
- rspec:19:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
neutral:OrdersController#index:/Users/rdodson/Developer/marketplaces/app/controllers/orders_controller.rb:2:a8c5f
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :index,
  s(:args),
  s(:send, nil, :render,
    s(:hash,
      s(:pair,
        s(:sym, :json),
        s(:send,
          s(:const, nil, :Order), :paginate,
          s(:send, nil, :pagination_attributes))))))
Unparsed Source:
def index
  render(json: Order.paginate(pagination_attributes))
end
Test Result:
- 2 @ runtime: 0.119466
  - rspec:18:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
  - rspec:19:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
Test Output:
end of file reached
-----------------------
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController#index]>
Integration:     Mutant::Integration::Rspec
Jobs:            1
Includes:        []
Requires:        []
Subjects:        1
Mutations:       21
Results:         21
Kills:           20
Alive:           1
Runtime:         3.01s
Killtime:        2.38s
Overhead:        26.59%
Mutations/s:     6.97
Coverage:        95.24%
Coverage report generated for RSpec to /Users/rdodson/Developer/marketplaces/coverage. 93 / 610 LOC (15.25%) covered.

I am pretty certain that my tests are not leaking data - my application is fairly simple, and does not use any unusual custom configuration of Rails, RSpec, and FactoryGirl, as far as I know. That said, I will try to reproduce this issue in a minimal, public repository and link to it here.

[rdodson41]

@mbj also, if you have a moment, would you mind explaining in further detail what you mean by "mutant invariants" and "invariant violation", as well as what kind of leaky spec setup you think could cause this?

[mbj]

@rdodson41 I need to write the invariants down in a central location referenced from the readme. I explained them often in slack and maybe even in GH comments.

But right now I cannot expand as I'm busy. Feel free to search GH / Slack. Sorry I know this a disgusting answer.

[rdodson41]

@mbj No worries, I understand completely; I will look into previous GitHub issues and comments. Per your request, I created a public repository of a minimal version of my application: https://github.com/rdodson41/mutant_factory_girl.

I just noticed something interesting: this issue occurs for me when I use Ruby v2.4.0 and v2.4.1, but not v2.4.2. The following is the output of bundle exec mutant --use rspec OrdersController under each of those three Ruby versions. I apologize for the long code blocks:

$ rvm use 2.4.0
Using /Users/rdodson/.rvm/gems/ruby-2.4.0
$ bundle exec mutant --use rspec OrdersController
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
(00/92) 100% - killtime: 0.00s runtime: 0.00s overhead: 0.00s
(43/92)  93% - killtime: 7.53s runtime: 1.01s overhead: -6.51s
(82/92)  95% - killtime: 15.10s runtime: 2.01s overhead: -13.08s
(88/92)  95% - killtime: 16.30s runtime: 3.02s overhead: -13.28s
OrdersController#id:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:16
- rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
- rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
- rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
neutral:OrdersController#id:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:16:f5d37
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :id,
  s(:args),
  s(:send,
    s(:send, nil, :params), :require,
    s(:sym, :id)))
Unparsed Source:
def id
  params.require(:id)
end
Test Result:
- 3 @ runtime: 0.16609
  - rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
  - rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
  - rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
Test Output:
end of file reached
-----------------------
OrdersController#index:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:2
- rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
- rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
neutral:OrdersController#index:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:2:59a4b
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :index,
  s(:args),
  s(:send, nil, :render,
    s(:hash,
      s(:pair,
        s(:sym, :json),
        s(:send,
          s(:const, nil, :Order), :paginate,
          s(:send, nil, :pagination_attributes))))))
Unparsed Source:
def index
  render(json: Order.paginate(pagination_attributes))
end
Test Result:
- 2 @ runtime: 0.162992
  - rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
  - rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
Test Output:
end of file reached
-----------------------
OrdersController#pagination_attributes:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:12
- rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
- rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
- rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
neutral:OrdersController#pagination_attributes:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:12:ace98
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :pagination_attributes,
  s(:args),
  s(:hash,
    s(:pair,
      s(:sym, :page),
      s(:send,
        s(:send, nil, :params), :[],
        s(:sym, :page))),
    s(:pair,
      s(:sym, :per_page),
      s(:send,
        s(:send, nil, :params), :[],
        s(:sym, :per_page)))))
Unparsed Source:
def pagination_attributes
  { page: params[:page], per_page: params[:per_page] }
end
Test Result:
- 3 @ runtime: 0.16451
  - rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
  - rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
  - rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
Test Output:
end of file reached
-----------------------
OrdersController#show:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:6
- rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
neutral:OrdersController#show:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:6:2da88
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :show,
  s(:args),
  s(:send, nil, :render,
    s(:hash,
      s(:pair,
        s(:sym, :json),
        s(:send,
          s(:const, nil, :Order), :find,
          s(:send, nil, :id))))))
Unparsed Source:
def show
  render(json: Order.find(id))
end
Test Result:
- 1 @ runtime: 0.190235
  - rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
Test Output:
end of file reached
-----------------------
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
Subjects:        4
Mutations:       92
Results:         92
Kills:           88
Alive:           4
Runtime:         3.02s
Killtime:        16.30s
Overhead:        -81.49%
Mutations/s:     30.49
Coverage:        95.65%
Coverage report generated for RSpec to /Users/rdodson/Developer/mutant-factory-girl/coverage. 9 / 23 LOC (39.13%) covered.

$ rvm use 2.4.1
Using /Users/rdodson/.rvm/gems/ruby-2.4.1
$ bundle exec mutant --use rspec OrdersController
warning: parser/current is loading parser/ruby24, which recognizes
warning: 2.4.0-compliant syntax, but you are running 2.4.1.
warning: please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
(00/92) 100% - killtime: 0.00s runtime: 0.00s overhead: 0.00s
(40/92)  93% - killtime: 7.03s runtime: 1.01s overhead: -6.02s
(75/92)  94% - killtime: 14.67s runtime: 2.02s overhead: -12.65s
(88/92)  95% - killtime: 17.37s runtime: 3.03s overhead: -14.34s
OrdersController#id:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:16
- rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
- rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
- rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
neutral:OrdersController#id:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:16:f5d37
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :id,
  s(:args),
  s(:send,
    s(:send, nil, :params), :require,
    s(:sym, :id)))
Unparsed Source:
def id
  params.require(:id)
end
Test Result:
- 3 @ runtime: 0.15252
  - rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
  - rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
  - rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
Test Output:
end of file reached
-----------------------
OrdersController#index:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:2
- rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
- rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
neutral:OrdersController#index:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:2:59a4b
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :index,
  s(:args),
  s(:send, nil, :render,
    s(:hash,
      s(:pair,
        s(:sym, :json),
        s(:send,
          s(:const, nil, :Order), :paginate,
          s(:send, nil, :pagination_attributes))))))
Unparsed Source:
def index
  render(json: Order.paginate(pagination_attributes))
end
Test Result:
- 2 @ runtime: 0.157153
  - rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
  - rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
Test Output:
end of file reached
-----------------------
OrdersController#pagination_attributes:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:12
- rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
- rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
- rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
neutral:OrdersController#pagination_attributes:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:12:ace98
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :pagination_attributes,
  s(:args),
  s(:hash,
    s(:pair,
      s(:sym, :page),
      s(:send,
        s(:send, nil, :params), :[],
        s(:sym, :page))),
    s(:pair,
      s(:sym, :per_page),
      s(:send,
        s(:send, nil, :params), :[],
        s(:sym, :per_page)))))
Unparsed Source:
def pagination_attributes
  { page: params[:page], per_page: params[:per_page] }
end
Test Result:
- 3 @ runtime: 0.190216
  - rspec:0:./spec/controllers/orders_controller_spec.rb:16/OrdersController#index without pagination attributes responds with orders
  - rspec:1:./spec/controllers/orders_controller_spec.rb:34/OrdersController#index with pagination attributes responds with orders
  - rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
Test Output:
end of file reached
-----------------------
OrdersController#show:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:6
- rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
neutral:OrdersController#show:/Users/rdodson/Developer/mutant-factory-girl/app/controllers/orders_controller.rb:6:2da88
--- Neutral failure ---
Original code was inserted unmutated. And the test did NOT PASS.
Your tests do not pass initially or you found a bug in mutant / unparser.
Subject AST:
s(:def, :show,
  s(:args),
  s(:send, nil, :render,
    s(:hash,
      s(:pair,
        s(:sym, :json),
        s(:send,
          s(:const, nil, :Order), :find,
          s(:send, nil, :id))))))
Unparsed Source:
def show
  render(json: Order.find(id))
end
Test Result:
- 1 @ runtime: 0.212928
  - rspec:2:./spec/controllers/orders_controller_spec.rb:57/OrdersController#show responds with the order
Test Output:
end of file reached
-----------------------
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
Subjects:        4
Mutations:       92
Results:         92
Kills:           88
Alive:           4
Runtime:         3.03s
Killtime:        17.37s
Overhead:        -82.56%
Mutations/s:     30.37
Coverage:        95.65%
Coverage report generated for RSpec to /Users/rdodson/Developer/mutant-factory-girl/coverage. 9 / 23 LOC (39.13%) covered.

$ rvm use 2.4.2
Using /Users/rdodson/.rvm/gems/ruby-2.4.2
$ bundle exec mutant --use rspec OrdersController
warning: parser/current is loading parser/ruby24, which recognizes
warning: 2.4.0-compliant syntax, but you are running 2.4.2.
warning: please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
(00/92) 100% - killtime: 0.00s runtime: 0.00s overhead: 0.00s
(25/92) 100% - killtime: 4.23s runtime: 1.00s overhead: -3.22s
(48/92) 100% - killtime: 8.77s runtime: 2.01s overhead: -6.76s
(72/92) 100% - killtime: 13.91s runtime: 3.01s overhead: -10.90s
(92/92) 100% - killtime: 17.75s runtime: 4.01s overhead: -13.74s
Mutant configuration:
Matcher:         #<Mutant::Matcher::Config match_expressions: [OrdersController]>
Integration:     Mutant::Integration::Rspec
Jobs:            8
Includes:        []
Requires:        []
Subjects:        4
Mutations:       92
Results:         92
Kills:           92
Alive:           0
Runtime:         4.01s
Killtime:        17.75s
Overhead:        -77.41%
Mutations/s:     22.94
Coverage:        100.00%
Coverage report generated for RSpec to /Users/rdodson/Developer/mutant-factory-girl/coverage. 9 / 23 LOC (39.13%) covered.

Let me know what you think. I might look into the Ruby changelog for anything that might have inadvertently fixed this issue in v2.4.2.

[rdodson41]

Similarly, this issue occurs in Ruby v2.3.4 but not in v2.3.5.

[mbj]

@mbj also, if you have a moment, would you mind explaining in further detail what you mean by "mutant invariants" and "invariant violation", as well as what kind of leaky spec setup you think could cause this?

Here is a good summary: https://github.com/mbj/mutant/issues/760#issuecomment-438034690

[mbj]

@rdodson41 I'll close this as it not had activity for a while, feel free to look at https://github.com/mbj/mutant/issues/760#issuecomment-438034690 for inspiration. If its still a problem feel free to re-open.