search

codeceptjs / CodeceptJS

Supercharged End 2 End Testing Framework for NodeJS
http://codecept.io
MIT License
4.11k stars 722 forks source link

Plugin "stepByStepReport" doesn't synchronize screenshots with steps #4405

Closed mirao closed 3 months ago

mirao commented 3 months ago

What are you trying to achieve?

A screenshot should be done after each step in the scenario - there are 4 steps in the scenario, so the screenshot should display a result after each of those 4 steps.

What do you get instead?

All screenshots are done at the very beginning of the test. E.g. you can't see there filling of the text in the search field or a result of the search query: record.zip

image

Provide console output if related. Use --verbose mode for more details.

$ codeceptjs run --verbose
***************************************
nodeInfo:  18.20.3
osInfo:  Linux 6.5 Ubuntu 22.04.4 LTS 22.04.4 LTS (Jammy Jellyfish)
cpuInfo:  (16) x64 AMD Ryzen 9 7940HS w/ Radeon 780M Graphics
chromeInfo:  126.0.6478.114
edgeInfo:  "N/A"
firefoxInfo:  undefined
safariInfo:  N/A
If you need more detailed info, just run this: npx codeceptjs info
***************************************
CodeceptJS v3.6.4 #StandWithUkraine
Using test root "/home/mirao/workspace/tmp"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport, tryTo, retryFailedStep, retryTo, eachElement

StepByStep --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something
 › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0000.png
 › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0001.png
 › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0002.png
 › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0003.png
    I am on page "https://codecept.io"
    I fill field "#algolia-search-input", "Step by Step"
    I press key "Enter"
    I see element "#stepbystepreport"
  ✔ OK in 1688ms

  OK  | 1 passed   // 2s
◉ Step-by-step preview: file:///home/mirao/workspace/tmp/output/records.html

Provide test source code if related

Feature('StepByStep');

Scenario('test something',  ({ I }) => {
    I.amOnPage("https://codecept.io")
    I.fillField("#algolia-search-input", "Step by Step")
    I.pressKey("Enter");
    I.seeElement("#stepbystepreport");
});

Details

import { setHeadlessWhen, setCommonPlugins } from '@codeceptjs/configure';
// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

// enable all common plugins https://github.com/codeceptjs/configure#setcommonplugins
setCommonPlugins();

export const config: CodeceptJS.MainConfig = {
  tests: './*_test.ts',
  output: './output',
  helpers: {
    Playwright: {
      browser: 'chromium',
      url: 'http://localhost',
      show: true
    }
  },
  plugins: {
    stepByStepReport: {
      enabled: true,
        deleteSuccessful: false,
      },
  },
  include: {
    I: './steps_file'
  },
  name: 'tmp'
}
mirao commented 3 months ago

It seems that CodeceptJS creates a screenshot when adding the scenario steps into a "recorder" and not when executing the steps.

mirao commented 3 months ago

Looks like broken by https://github.com/codeceptjs/CodeceptJS/pull/4385 . It works for me with 3.6.2 and doesn't work with 3.6.3 and 3.6.4

3.6.2:

StepByStep --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something
    I am on page "https://codecept.io"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0000.png
    I fill field "#algolia-search-input", "Step by Step"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0001.png
    I press key "Enter"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0002.png
    I see element "#stepbystepreport"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0003.png
  ✔ OK in 2971ms

  OK  | 1 passed   // 4s

CC: @kobenguyent

kobenguyent commented 3 months ago

hey @mirao would you mind trying this test version codeceptjs@3.6.5-beta.1? Hope this resolves the issue.

  Github login
    I am on page "https://github.com"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0000.png
    I click "Sign in"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0001.png
    I fill field "Username", "davert"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0002.png
    I fill field "Password", "123345"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0003.png
    I click "Login"
    [1]  Error (Non-Terminated) | Error: Clickable element "Login" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
    [1] Error | Error: Clickable element "Login" was not found by text|CSS|XPath undefined...
    [1] <teardown>  Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/Github_login.failed.png
 › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_db74bd544537c0c200ea296ba40d5f69c9ecd703bf7a4677ca6ec1e9acad5a59/0004.png
  ✖ FAILED in 1940ms
mirao commented 3 months ago

@kobenguyent It doesn't work for me with codeceptjs@3.6.5-beta.1

CodeceptJS v3.6.5-beta.1 #StandWithUkraine
Using test root "/home/mirao/workspace/codeceptjs/tests/my"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport

My --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something
    I am on page "https://codecept.io"
    I fill field "#algolia-search-input", "Step by Step"
    I press key "Enter"
    I see element "#stepbystepreport"
  ✔ OK in 4011ms

And I'm getting only record_hash/index.html without screenshots and without records.html.

This is with 3.6.4:

CodeceptJS v3.6.4 #StandWithUkraine
Using test root "/home/mirao/workspace/codeceptjs/tests/my"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport

My --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
  test something
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/my/output/record_634ff7f892cec2ec4613ffd413788cd8b96be53470a5ea6834e2091e9416a588/0000.png
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/my/output/record_634ff7f892cec2ec4613ffd413788cd8b96be53470a5ea6834e2091e9416a588/0001.png
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/my/output/record_634ff7f892cec2ec4613ffd413788cd8b96be53470a5ea6834e2091e9416a588/0002.png
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/my/output/record_634ff7f892cec2ec4613ffd413788cd8b96be53470a5ea6834e2091e9416a588/0003.png
    I am on page "https://codecept.io"
    I fill field "#algolia-search-input", "Step by Step"
    I press key "Enter"
    I see element "#stepbystepreport"
  ✔ OK in 3081ms

  OK  | 1 passed   // 4s
◉ Step-by-step preview: file:///home/mirao/workspace/codeceptjs/tests/my/output/records.html
kobenguyent commented 3 months ago

@mirao sorry, wrong push, please try with this test version codeceptjs@3.6.5-beta.2

mirao commented 3 months ago

@kobenguyent It works. But I noticed that if you have BeforeSuite() in a test, it fails :bug: (it didn't fail with 3.6.4)

Test

Feature("StepByStep");

BeforeSuite(({ I }) => {
    I.say("Before Suite");
});

Scenario("test something", ({ I }) => {
    I.amOnPage("https://codecept.io");
    I.fillField("#algolia-search-input", "Step by Step");
    I.pressKey("Enter");
    I.seeElement("#stepbystepreport");
});

output

CodeceptJS v3.6.5-beta.2 #StandWithUkraine
Using test root "/home/mirao/workspace/codeceptjs/tests/my"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport, tryTo, retryFailedStep, retryTo, eachElement

StepByStep --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
    --- STARTED "before all" hook: BeforeSuite for "test something" ---
    I say "Before Suite"
    [1]  Error (Non-Terminated) | TypeError: Cannot read properties of undefined (reading 'toString') | (err) => { step.status = 'failed'; step.endTime = ...
    [1] Error | TypeError: Cannot read properties of undefined (reading 'toString') (e) => { const err = recorder.getAsyncErr() === nu...
 › <screenshotOnFail> BeforeSuite/AfterSuite do not have any access to the browser, hence it could not take screenshot.
 › <stepByStepReport> BeforeSuite/AfterSuite do not have any access to the browser, hence it could not take screenshot.
    [1] <teardown>  Stopping recording promises
  ✖ FAILED in 8ms

    [2]  Starting recording promises

-- FAILURES:

  1) StepByStep
       "before all" hook: BeforeSuite for "test something":
     Cannot read properties of undefined (reading 'toString')
      at persistStep (node_modules/.pnpm/codeceptjs@3.6.5-beta.2_@babel+core@7.24.7_@babel+preset-env@7.24.7_@babel+core@7.24.7__@jest_pddntehogpft3mtwd3fjgeop4e/node_modules/codeceptjs/lib/plugin/stepByStepReport.js:158:63)
      at /home/mirao/workspace/codeceptjs/tests/my/node_modules/.pnpm/codeceptjs@3.6.5-beta.2_@babel+core@7.24.7_@babel+preset-env@7.24.7_@babel+core@7.24.7__@jest_pddntehogpft3mtwd3fjgeop4e/node_modules/codeceptjs/lib/plugin/stepByStepReport.js:110:60

  Scenario Steps:
  - I.say("Before Suite") at Context.<anonymous> (./StepByStep_test.ts:4:7)

  FAIL  | 0 passed, 1 failed   // 211ms
kobenguyent commented 3 months ago

@mirao one more try with codeceptjs@3.6.5-beta.3, please.

login --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
    --- STARTED "before all" hook: BeforeSuite for "test something" ---
    I say "Before Suite"
   Before Suite
    --- ENDED "before all" hook: BeforeSuite for "test something" ---
  test something
    I am on page "https://codecept.io"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_e85e066f8f5deb624c13c7c305bd271329ed56a3e1b465362fa6c3d1f5996362/0000.png
    I fill field "#algolia-search-input", "Step by Step"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_e85e066f8f5deb624c13c7c305bd271329ed56a3e1b465362fa6c3d1f5996362/0001.png
    I press key "Enter"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_e85e066f8f5deb624c13c7c305bd271329ed56a3e1b465362fa6c3d1f5996362/0002.png
    I see element "#stepbystepreport"
    › Screenshot is saving to /Users/t/Desktop/projects/codeceptjs-playwright-fun/output/record_e85e066f8f5deb624c13c7c305bd271329ed56a3e1b465362fa6c3d1f5996362/0003.png
  ✔ OK in 3318ms

  OK  | 1 passed   // 5s
◉ Step-by-step preview: file:///Users/t/Desktop/projects/codeceptjs-playwright-fun/output/records.html
mirao commented 3 months ago

It works well with codeceptjs@3.6.5-beta.3. Only issue is that AfterSuite() isn't omitted from the plugin. It doesn't cause a failure on exit, but it looks nasty. The issue is even in 3.6.4 Could you fix it, please?

test:

Feature("StepByStep");

BeforeSuite(({ I }) => {
  I.say("Before Suite");
});

Scenario("test something", ({ I }) => {
  I.amOnPage("https://codecept.io");
  I.fillField("#algolia-search-input", "Step by Step");
  I.pressKey("Enter");
  I.seeElement("#stepbystepreport");
});

AfterSuite(({ I }) => {
  I.say("After Suite");
});

output:

CodeceptJS v3.6.5-beta.3 #StandWithUkraine
Using test root "/home/mirao/workspace/tmp"
Helpers: Playwright
Plugins: screenshotOnFail, stepByStepReport, tryTo, retryFailedStep, retryTo, eachElement

StepByStep --
    [1]  Starting recording promises
    Timeouts: 
 › [Session] Starting singleton browser session
    --- STARTED "before all" hook: BeforeSuite for "test something" ---
    I say "Before Suite"
   Before Suite
    --- ENDED "before all" hook: BeforeSuite for "test something" ---
  test something
    I am on page "https://codecept.io"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0000.png
    I fill field "#algolia-search-input", "Step by Step"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0001.png
    I press key "Enter"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0002.png
    I see element "#stepbystepreport"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0003.png
  ✔ OK in 4392ms

    --- STARTED "after all" hook: AfterSuite for "test something" ---
    I say "After Suite"
    › Screenshot is saving to /home/mirao/workspace/tmp/output/record_c207c723ddc6ff2089642ffbbf5963a68a859f6ffd2558cea60869b5d0bf7dfa/0004.png
    › <Can't save step screenshot: Error: page.screenshot: Target page, context or browser has been closed> 
   After Suite
    --- ENDED "after all" hook: AfterSuite for "test something" ---

  OK  | 1 passed   // 5s
◉ Step-by-step preview: file:///home/mirao/workspace/tmp/output/records.html
kobenguyent commented 3 months ago

@mirao let's try with codeceptjs@3.6.5-beta.4.

mirao commented 3 months ago

@kobenguyent Works well now, thanks.

One more issue: When you have two scenarios and the 1st one fails, then the plugin stops creating screenshots for the 2nd scenario and finally records.html isn't created :bug: Could you look at it? Or may I create a separate issue for it? The bug is even in 3.6.4

test:

Feature("StepByStep");

BeforeSuite(({ I }) => {
    I.say("Before Suite");
});

Scenario("test something 1", ({ I }) => {
    I.amOnPage("https://codecept.io");
    I.fillField("#blabla", "Step by Step"); // A typo in element
    I.pressKey("Enter");
    I.seeElement("#stepbystepreport");
});

Scenario("test something 2", ({ I }) => {
    I.amOnPage("https://codecept.io");
    I.fillField("#algolia-search-input", "Step by Step");
    I.pressKey("Enter");
    I.seeElement("#stepbystepreport");
});

AfterSuite(({ I }) => {
    I.say("After Suite");
});

output:

StepByStep --
    [1]  Starting recording promises
    Timeouts: 600
 › [Session] Starting singleton browser session
 › Open login screen
    --- STARTED "before all" hook: BeforeSuite for "test something 1" ---
    I say "Before Suite"
   Before Suite
    --- ENDED "before all" hook: BeforeSuite for "test something 1" ---
  test something 1
 › Test Timeout: 600s
    I am on page "https://codecept.io"
    › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/s4h-web/output/record_0fc08ef3432d0381979450d4c2fa03d13b0844e2d6d8a7b6ec83da5cc693519e/0000.png
    I fill field "#blabla", "Step by Step"
    [1]  Error (Non-Terminated) | Error: Field "#blabla" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
    [1]  Error (Non-Terminated) | Error: Field "#blabla" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
    [1]  Error (Non-Terminated) | Error: Field "#blabla" was not found by text|CSS|XPath | (err) => { step.status = 'failed'; step.endTime = ...
    [1] Error | Error: Field "#blabla" was not found by text|CSS|XPath undefined...
    [1] <teardown>  Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/s4h-web/output/test_somet_1719563268.failed.png
 › Screenshot is saving to /home/mirao/workspace/codeceptjs/tests/s4h-web/output/record_0fc08ef3432d0381979450d4c2fa03d13b0844e2d6d8a7b6ec83da5cc693519e/0001.png
  ✖ FAILED in 2045ms

    [2]  Starting recording promises
  test something 2
 › Test Timeout: 600s
    I am on page "https://codecept.io"
    I fill field "#algolia-search-input", "Step by Step"
    I press key "Enter"
    I see element "#stepbystepreport"
  ✔ OK in 2133ms

    --- STARTED "after all" hook: AfterSuite for "test something 2" ---
    I say "After Suite"
   After Suite
    --- ENDED "after all" hook: AfterSuite for "test something 2" ---

-- FAILURES:

  1) StepByStep
       test something 1:
     Field "#blabla" was not found by text|CSS|XPath
      at new ElementNotFound (/home/mirao/workspace/codeceptjs/node_modules/.pnpm/codeceptjs@3.6.5-beta.4_patch_hash=eotzuczidxj7tycvn36zpciqru_@babel+core@7.24.7_@babel+prese_tysehjua4vp7stc2brmcbmndoe/node_modules/codeceptjs/lib/helper/errors/ElementNotFound.js:15:11)
      at assertElementExists (/home/mirao/workspace/codeceptjs/node_modules/.pnpm/codeceptjs@3.6.5-beta.4_patch_hash=eotzuczidxj7tycvn36zpciqru_@babel+core@7.24.7_@babel+prese_tysehjua4vp7stc2brmcbmndoe/node_modules/codeceptjs/lib/helper/Playwright.js:3714:11)
      at Playwright.fillField (/home/mirao/workspace/codeceptjs/node_modules/.pnpm/codeceptjs@3.6.5-beta.4_patch_hash=eotzuczidxj7tycvn36zpciqru_@babel+core@7.24.7_@babel+prese_tysehjua4vp7stc2brmcbmndoe/node_modules/codeceptjs/lib/helper/Playwright.js:1741:5)

  Scenario Steps:
  - I.fillField("#blabla", "Step by Step") at Test.<anonymous> (./StepByStep_test.ts:9:7)
  - I.amOnPage("https://codecept.io") at Test.<anonymous> (./StepByStep_test.ts:8:7)

  Artifacts:
  - screenshots: /home/mirao/workspace/codeceptjs/tests/s4h-web/output/record_0fc08ef3432d0381979450d4c2fa03d13b0844e2d6d8a7b6ec83da5cc693519e/0000.png
  - screenshot: /home/mirao/workspace/codeceptjs/tests/s4h-web/output/test_somet_1719563268.failed.png

  FAIL  | 1 passed, 1 failed   // 6s
mirao commented 3 months ago

One more issue: When you have two scenarios and the 1st one fails, then the plugin stops creating screenshots for the 2nd scenario and finally records.html isn't created 🐛

Well, it seems to be by design. disableScreenshotOnFail is true by default. If I set it to false then it works as expected. https://codecept.io/plugins/#stepbystepreport

mirao commented 3 months ago

@kobenguyent So you can close the task, everything seems to be solved. Thanks.

kobenguyent commented 3 months ago

@mirao thanks for your testing!