Processing example doesn't work properly with aggregate_failures (sets the test status as Untested or Passed)

[matisnape]

So I found out that when I have flag :aggregate_failures set to true, the processing example in after :example doesn't really work - the example is set as untested and when I try to debug, the example object has :execution_result only with attribute @started_at and there's no exception message. In other words, it looks like the first expect already concludes the example for processing for Testrail, even though it didn't really finish running.

Example:

      scenario 'in edit job', testrail_id: 5790 do
        expect(edit_job_page.question_form).to have_added_questions(count: 2, text: video_question)
        expect(edit_job_page.question_form).to have_content "Required"

        edit_job_page.go_to_job_posting
        edit_job_page.go_to_edit_application_form

        expect(edit_job_page.question_form).to have_added_questions(count: 2, text: video_question)
        expect(edit_job_page.question_form).to have_content "Required22"
      end

In this spec I have 4 expects and 3 are failing:

  1) Screening questions adding can add video answer question in edit job
     Got 3 failures:

     1.1) Failure/Error: expect(edit_job_page.question_form).to have_added_questions(count: 2, text: video_question)
            expected #has_added_questions?({:count=>2, :text=>"Example video answer questions"}) to return true, got false
          # ./spec/jobs/screening_questions_spec.rb:256:in `block (4 levels) in <top (required)>'
          # ./spec/spec_helper.rb:39:in `block (2 levels) in <top (required)>'

     1.2) Failure/Error: expect(edit_job_page.question_form).to have_added_questions(count: 2, text: video_question)
            expected #has_added_questions?({:count=>2, :text=>"Example video answer questions"}) to return true, got false
          # ./spec/jobs/screening_questions_spec.rb:261:in `block (4 levels) in <top (required)>'
          # ./spec/spec_helper.rb:39:in `block (2 levels) in <top (required)>'

     1.3) Failure/Error: expect(edit_job_page.question_form).to have_content "Required22"
            expected to find text "Required22" in "Add new"
          # ./spec/jobs/screening_questions_spec.rb:262:in `block (4 levels) in <top (required)>'
          # ./spec/spec_helper.rb:39:in `block (2 levels) in <top (required)>'

Digging in the Rspec core documentation I found that aggregate_failures is like a couple examples that are later normalized into single backtrace, but I don't know how to make processing wait for all failures to be normalized.

Do you have any idea how to do that maybe? :)

[matisnape]

Basically, I rewrote the integration a bit - I now use a reporter listener instead of before/after hooks :)

[ilyazub]

Linking GitHub issue of rspec-core that clarifies the execution flow: https://github.com/rspec/rspec-core/issues/2289