in fully_formatted_failed_examples - undefined method 'fully_formatted' for RSpec::Core::Notifications::ExampleNotification

[Nitrodist]

Subject of the issue

I have flaky tests so in the process of creating a second test runner in CircleCI to run just the flaky tests, I've encountered this stack trace after using --tag flaky in one set of tests:

/home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/notifications.rb:114:in `block in fully_formatted_failed_examples': undefined method `fully_formatted' for #<RSpec::Core::Notifications::ExampleNotification:0x0000558ccbb56618> (NoMethodError)

Full trace:

#!/bin/bash -eo pipefail
bundle exec rspec --format RspecJunitFormatter --out tmp/test-results/rspec.xml --format progress --tag flaky $(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=name)
Run options: include {:flaky=>true}
[Zonebie] Setting timezone: ZONEBIE_TZ="Skopje"

🐢  Precompiling assets.
Puma starting in single mode...
* Puma version: 5.6.4 (ruby 2.7.4-p191) ("Birdie's Version")
*  Min threads: 4
*  Max threads: 4
*  Environment: test
*          PID: 5615
* Listening on http://127.0.0.1:38825
Use Ctrl-C to stop
............F

bundler: failed to load command: rspec (/home/circleci/project/vendor/bundle/ruby/2.7.0/bin/rspec)
Coverage report generated for RSpec to /home/circleci/project/coverage. 5674 / 14251 LOC (39.81%) covered.
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected
Traceback (most recent call last):
    33: from /home/circleci/.rubygems/bin/bundle:23:in `<main>'
    32: from /home/circleci/.rubygems/bin/bundle:23:in `load'
    31: from /home/circleci/.rubygems/gems/bundler-2.3.8/exe/bundle:36:in `<top (required)>'
    30: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
    29: from /home/circleci/.rubygems/gems/bundler-2.3.8/exe/bundle:48:in `block in <top (required)>'
    28: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/cli.rb:25:in `start'
    27: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
    26: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/cli.rb:31:in `dispatch'
    25: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
    24: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    23: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
    22: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/cli.rb:484:in `exec'
    21: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/cli/exec.rb:23:in `run'
    20: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/cli/exec.rb:58:in `kernel_load'
    19: from /home/circleci/.rubygems/gems/bundler-2.3.8/lib/bundler/cli/exec.rb:58:in `load'
    18: from /home/circleci/project/vendor/bundle/ruby/2.7.0/bin/rspec:23:in `<top (required)>'
    17: from /home/circleci/project/vendor/bundle/ruby/2.7.0/bin/rspec:23:in `load'
    16: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/exe/rspec:4:in `<top (required)>'
    15: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/runner.rb:45:in `invoke'
    14: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/runner.rb:71:in `run'
    13: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/runner.rb:89:in `run'
    12: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/runner.rb:115:in `run_specs'
    11: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/reporter.rb:76:in `report'
    10: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/reporter.rb:174:in `finish'
     9: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/reporter.rb:194:in `close_after'
     8: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/reporter.rb:178:in `block in finish'
     7: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/reporter.rb:208:in `notify'
     6: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/reporter.rb:208:in `each'
     5: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/reporter.rb:209:in `block in notify'
     4: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/formatters/base_text_formatter.rb:32:in `dump_failures'
     3: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/notifications.rb:113:in `fully_formatted_failed_examples'
     2: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/notifications.rb:113:in `each_with_index'
     1: from /home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/notifications.rb:113:in `each'
/home/circleci/project/vendor/bundle/ruby/2.7.0/gems/rspec-core-3.12.0/lib/rspec/core/notifications.rb:114:in `block in fully_formatted_failed_examples': undefined method `fully_formatted' for #<RSpec::Core::Notifications::ExampleNotification:0x0000558ccbb56618> (NoMethodError)

Exited with code exit status 1
CircleCI received exit code 1

Your environment

• Ruby version: 2.7.4
• rspec-core version: rspec-core (3.12.0) (also fails in 3.9.2 and 3.9.3)

Steps to reproduce

Huh, this is actually kinda really hard to reproduce because it's only happening CircleCI! Here is the command we're using to run the tests:

      - run: bundle exec rspec --format RspecJunitFormatter --out tmp/test-results/rspec.xml --format progress --tag flaky $(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=name)

Expected behavior

I see the failure instead of an exception being raised from within the rspec-core internals.

Actual behavior

Exception internal to rspec-core is being raised.

[pirj]

So the failure is a RSpec::Core::Notifications::ExampleNotification. failure_notifications map through failed_examples with ExampleNotification.for. new is private in ExampleNotification. The only place where it's called is:

return new(example) unless execution_result.status == :pending || execution_result.status == :failed

and this is when the example is neither failed nor pending. But @reporter.failed_examples should only contain failed examples 🤔

Would you fork rspec-core, add a bogus missing method to that class, and use it with your ci to see what happens? Are those 13 examples all of your flaky examples? Does it abruptly exit on the first failed example, or is it just a coincidence that the failed example was the last out of 13? What if you exclude the JUnit formatter? What if you run just those flaky examples one by one? Do you have that many flaky examples that you have to provide a list of files to rspec (the --split-by=name part)?