pirj
commented
2 years ago So the failure is a RSpec::Core::Notifications::ExampleNotification. failure_notifications map through failed_examples with ExampleNotification.for. new is private in ExampleNotification. The only place where it's called is:
return new(example) unless execution_result.status == :pending || execution_result.status == :failedand this is when the example is neither failed nor pending.
But @reporter.failed_examples should only contain failed examples 🤔
Would you fork rspec-core, add a bogus missing method to that class, and use it with your ci to see what happens?
Are those 13 examples all of your flaky examples?
Does it abruptly exit on the first failed example, or is it just a coincidence that the failed example was the last out of 13?
What if you exclude the JUnit formatter?
What if you run just those flaky examples one by one?
Do you have that many flaky examples that you have to provide a list of files to rspec (the --split-by=name part)?
JonRowe
Subject of the issue
I have flaky tests so in the process of creating a second test runner in CircleCI to run just the flaky tests, I've encountered this stack trace after using
--tag flakyin one set of tests:Full trace:
Your environment
Steps to reproduce
Huh, this is actually kinda really hard to reproduce because it's only happening CircleCI! Here is the command we're using to run the tests:
Expected behavior
I see the failure instead of an exception being raised from within the rspec-core internals.
Actual behavior
Exception internal to rspec-core is being raised.