SeleniumHQ / selenium-ide

Open Source record and playback test automation for the web.
https://selenium.dev/selenium-ide/
Apache License 2.0
2.81k stars 765 forks source link

Output failed command/step for selenium-side-runner in summary #1233

Open ghost opened 3 years ago

ghost commented 3 years ago

πŸš€ Feature Proposal

Output failed command number for selenium-side-runner

Motivation

Other tools used for End User Monitoring usually output the failed step or command number. selenium-side-runner currently outputs the failure in a "code-like" format.

Making selenium-side-runner more friendly when performing EUM, an output that includes failed step/command number would be beneficial to identify which part of the SIDE project failed to complete successfully.

Current jest output is 75% of the way there, it just needs another field showing which numbered command (or description of command) failed.

Example

My example is separated out of the code block and in bold. The text is just an example, but at a minimum the failed command/step number, and additionally the command set (i.e verify text -> element -> value expected).

The audience of the extra output would be non-technical users to be able to interpret the failure in layman's terms with minimal requirement to understand the technical flow of the test.

$ selenium-side-runner -s http://localhost:4444/wd/hub myproject.side --output-directory test --output-format junit
info:    Running myproject.side
added 26 packages from 24 contributors and audited 26 packages in 2.297s
found 0 vulnerabilities

 FAIL  ./DefaultSuite.test.js (14.377s)
  Default Suite
    βœ• MyTest (11494ms)

  ● Default Suite β€Ί MyTest

    expect(received).toHaveText(expected)

    Expected value to be (using Object.is):
      "does not exist"
    Received:
      "this text is present"

      31 |   await driver.wait(until.elementLocated(By.xpath(`//div[@id=\'copyright-wrapper\']/div/div/div/div`)), 30000);
      32 |   await driver.wait(until.elementLocated(By.css(`.col-md-6 > div`)), configuration.timeout);
    > 33 |   await expect(driver.findElement(By.css(`.col-md-6 > div`))).resolves.toHaveText(`does not exist`);
         |                                                                        ^
      34 | }
      35 | module.exports = tests;

      at Object.toHaveText (../../../usr/local/lib/node_modules/selenium-side-runner/node_modules/expect/build/index.js:202:20)
      at Object.<anonymous>.tests.MyTest (commons.js:33:72)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total

Failed Step: 8 (verify text -> css=.col-md-6 > div; value="does not exist")

Snapshots:   0 total
Time:        14.598s, estimated 15s
Ran all test suites.
ghost commented 3 years ago

To counter myself, I could do something like grep the output for the > and then use that as the error report, however out of context (without knowing the rest of the script) a layman might not understand what it means.