aggregate_failures hides exceptions in after block

[RodneyU215]

Hi folks,

Some coworkers(@sfbrian & @claudiajf) and I stumbled upon what seems like a bug when using aggregate_failures. In our tests we have RSpec after hooks setup that check for example.exception which works normally. However when a spec has the aggregate_failures tag and it throws an exception, example.exception is nil.

PR #2288 has a spec that demonstrates this behavior.

@myronmarston when using aggregate_failures is there another way we should be checking for exceptions after an example other than this?

RSpec.configure do |config|
  config.after do |example|
    # Both return true with aggregate_failures
    example.exception.nil?
    example.execution_result.exception.nil?
  end
end

[myronmarston]

This is by design. Remember that an after hook -- while a separate block of code from it -- is really part of the example. Failures in an after hook will make the example fail. Since it could fail, aggregate_failures is still waiting to see if there are any more exceptions to include in its single, aggregate exception, so the example exception (if any) hasn't been assigned at that point.

In general, you can't count on the final result of an example being available in an after hook since it the hook is really part of the example. Instead, you should wire up a listener to be notified after each example finishes and you can check the result there:

class ExampleListener
  def example_finished(notification)
    # you can access notification.example.exception here
  end
end

RSpec.configure do |c|
  c.reporter.register_listener ExampleListener.new, :example_finished
end

[RodneyU215]

Thanks for the quick reply! I'll give it a try!

[RodneyU215]

Hey @myronmarston I appreciate the work around. It works for checking the exception! Would you happen to also know why notification.example.execution_result.finished_at and notification.example.execution_result.status both return nil? My expectations was that the reporter ran after record_finished was called.

[myronmarston]

That surprises me, too. Can you put together an reproducible example demonstrating the issue?

[RodneyU215]

Sure thing! I'll work on that and I'll open a new issue to track it.

[MaicolBen]

What if you want to use around ? I want to catch the exception and retry the example. Also, it seems you cannot register for specific tags.

[pirj]

@MaicolBen

What if you want to use around ? I want to catch the exception and retry the example.

It seems to be possible according to https://github.com/NoRedInk/rspec-retry/issues/72#issuecomment-509770334

Also, it seems you cannot register for specific tags.

Not sure what you mean exactly, can you please elaborate?

[MaicolBen]

Not sure what you mean exactly

Achieve the next code with register:

config.around :each, type: :feature do |ex|
   ex.run_with_retry retry: 3
 end

It seems to be possible according to

I came from there, for now, we don't know is possible until we see some code. I tried myself and failed, so I'm going to remove aggregate_failures

[JonRowe]

Around hooks are also part of the example, so the same problem is likely to exist, I wonder how we can expose that aggregate failures is in use and some how allow access to see if any exceptions have occurred so far (and potentially clear them), as thats all whats needed for rspec-retry's purposes...

config.around :each, type: :feature do |ex|
  ex.run_with_retry retry: 3
end

This code should work, as long as it's being called in a required configuration block before the spec is run...