rescue ExpectationNotMetError is ignored in around block

[iftheshoefritz]

Subject of the issue

I've got a rescue block around example.run in an around hook. The test fails and exits without running the code in the rescue block.

Context: I'm trying to create a label that when applied will mark an example pending only if it fails; i.e. if it passes I do not want it to make the entire suite fail the way pending does.

Your environment

• Ruby version:

  ⚡  ruby -v
  ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [arm64-darwin21]

• rspec-core version:

  ⚡  rspec -v
  Running via Spring preloader in process 56519
  RSpec 3.11
  - rspec-core 3.11.0
  - rspec-expectations 3.11.0
  - rspec-mocks 3.11.1
  - rspec-rails 5.1.2
  - rspec-support 3.11.0

Steps to reproduce

I based this around hook on an old stack overflow post

# spec/support/rails_helper.rb
RSpec.configure do |config|
    config.around(:each, flaky: true) do |example|
      example.run
    rescue RSpec::Expectations::ExpectationNotMetError => e
      pending "This test failed an expect clause, but it is marked with :flaky"
      raise e
  end
  ...
end

# spec/models/branch_spec.rb
it "test with flaky label", flaky: true do
  expect(true).to be false
end

Expected behaviour

I expect the failed expect call to be caught, the test to be marked pending, and then the failure to be raised again.

Actual behavior

The rescue block never executes and I get the standard failure:

Failures:

  1) test with flaky label fails
     Failure/Error: expect(true).to be false

       expected false
            got true
     # ./spec/models/branch_spec.rb:60:in `block (3 levels) in <main>'
     # ./spec/rails_helper.rb:29:in `block (2 levels) in <main>'
     # -e:1:in `<main>'

Happy to hear other approaches for achieving this goal if hooks just don't work this way.

[pirj]

There's no obvious workaround for this that comes to mind. Unless re-running the whole suite without flaky ones is an option.

Your use case resonates with my experience, I'd much rather prefer marking specs as flaky and fix them in a bulk rather than in scope of an unrelated PR with red CI due to flakiness.

@JonRowe Would it make sense to introduce a :flaky metadata handler/example alias, similar to pending/skip?

Example result	pending	skip	flaky
pass	failure	-	pass
fail	pass	-	pass

[JonRowe]

Have you checked you've setup the hook properly? i.e. that its actually running at all?

[iftheshoefritz]

The hook runs, but the test fails without reaching the rescue block.

[pirj]

This is as expected, we catch those exceptions deeper and only re-raise them in matchers that include expectations (see https://github.com/rspec/rspec-expectations/pull/892).

[JonRowe]

You should be able to check the examples run status to see if it has failed.