BDD skips tests in feature file if background fails in one test

[ktryniszewski-mdsol]

What are you trying to achieve?

Run a feature file without failing fast and skipping tests if background fails.

What do you get instead?

If you have a feature file with 5 scenarios and a Background. If a background step fails on scenario 3, then scenario 4 and 5 are marked as skipped. As we know sometimes a step fails because of timing issues or a page fails to load, this doesnt mean the background will fail each time. Isn't a Background treated as a beforeEach in mocha anyway? So it should run independently for each scenario.

Provide console output if related. Use --verbose mode for more details.

CodeceptJS v3.0.7
Helpers: Playwright, MochiavelliHelper, PlaywrightHelper, ChaiWrapper
Plugins:

Failing Background --
    [1]  Starting recording promises
 › [Session] Starting singleton browser session
  Search for something @test1
    Given I am on page "https://google.com"
      I am on page "https://google.com"
      › [Browser:Error]
      › [Browser:Error]
      › [Browser:Error]
    Then I should see "Fail test here"
      I see "Fail test here"
    [1] Error | Error
    [1] Error | Error
    [1] Starting <teardown> session
    [1] <teardown> Stopping recording promises
  ✖ FAILED in 23592ms

    [2]  Starting recording promises
 › [Session] cleaning cookies and localStorage
  S Search for something else @test2

-- FAILURES:

  1) Failing Background
       "before each" hook: Before for "Search for something @test1":

      expected web application to include "Fail test here"
      + expected - actual

      -About
      -Store
      -GmailImages
      -Sign in
      -
      -
      -
      -Celebrating the diverse stories of Asian and Pacific Islander communities
      -Advertising
      -Business
      ---( 6 lines more )---
      +Fail test here

  Scenario Steps:
  - I.see("Fail test here") at ./step_definitions/mock.steps.js:148:5
  - I.amOnPage("https://google.com") at ./step_definitions/mock.steps.js:144:5

  FAIL  | 0 passed, 1 failed, 1 skipped   // 24s

Provide test source code if related

steps:

Given('I am on page {string}', (url) => {
  I.amOnPage(url);
});

Then('I should see {string}', (text) => {
  I.see(text);
});

When('I fill in {string} in the search field', (text) => {
  I.fillField('.gLFyf.gsfi', text);
  I.pressKey('Enter');
});

feature file

Feature: Failing Background

  Background: Test
    Given I am on page "https://google.com"
    Then I should see "Fail test here"

  @test1
  Scenario: Search for something
    When I fill in "Test" in the search field

  @test2
  Scenario: Search for something else
    When I fill in "Google" in the search field

Details

• CodeceptJS version: 3.0.7
• NodeJS Version: v14.15.1
• Operating System: osx 10.15
• playwright 1.10

[jakoviktor]

You have no idea how grateful i am for your comment. Never would have guessed that codeceptjs had this strange mechanism.

[ktryniszewski-mdsol]

You have no idea how grateful i am for your comment. Never would have guessed that codeceptjs had this strange mechanism.

Yea there seems to be an underlying issue with how backgrounds are mapped to mocha's internal beforeEach. From current behavior it seems like they are being executed as beforeAll where it is only supposed to run once before a series of tests.

What is also odd, none of the logic you implement in a before hook is accessible in the background https://codecept.io/bdd/#before Meaning that retries wont be performed on background steps. Or if you try to setup a state object, it wont be accessible in the background steps.

Seems like a major flaw in how its currently implemented. I tried stepping through the code but didnt have enough time to investigate further

[DavertMik]

Thanks, looks like an severe bug. Not sure I will get time to check it out but any help is highly aporeciated

[DenisFominykh]

Getting the same behaviour with Playwright

[vtachkov]

We're running into the same issue as well, but in our case we have an error being thrown inside of a Before block. The same result happens where subsequent tests in the file are skipped.

[DavertMik]

Let me just clarify that Background is indeed executed as beforeEach:

See lib/interfaces/gherkin.js:101