[Bug]: Inconsistent Error Reporting in Test Steps Affecting JUnit Report Annotations

symon-skelly1991 commented 7 months ago

Version

1.41.2

Steps to reproduce

clone git repo:

https://github.com/symon-skelly1991/playwright-bug/tree/main
Run the tests with 1 - 10 workers, it doesn't matter how many. (npx playwright test --workers 10 (Powsershell))
Open the junit-report.xml file inside of testsrail-results

Expected behavior

All errors encountered during test steps should consistently be captured in testInfo._steps. Consequently, all relevant details, including errors, should be correctly annotated in the JUnit reports for each test, regardless of the test suite's execution context.

Actual behavior

The error details in testInfo._steps are inconsistently captured. As a result, the JUnit reports are missing crucial error information for some failing tests while showing complete details for others under similar failure conditions. This inconsistency is not related to the number of workers.

Some tests will have type="FAILURE", but all steps for the failed test have 'passed', this is because the 'error' object is for some reason missing from test._steps.

Additional context

Inconsistent error reporting in test steps has been identified as causing issues with JUnit report annotations. Specifically, there are instances where, despite test failures, the error details are not consistently captured in the test steps obtained from testInfo._steps. This inconsistency leads to incomplete annotations in the JUnit reports, particularly when running the full test suite. The issue appears randomly: sometimes the errors are properly included in the steps, while at other times, the steps lack the error details despite similar conditions and failures.

If the test.afterEach is inside of each spec.ts file, the issue does not occur and all tests contain the correct steps / errors / annotations as expected.

Environment

OS: Windows 11 / MAC 10.13
Node: 20.11.5
NPM: 10.2.5

dgozman commented 7 months ago

@symon-skelly1991 I tried running your repro, but I am not sure how to check that something does not work. In the junit-report.xml file I see testrail_result_step annotations all over the place, so not sure whether it worked or not. Please explain in details what's wrong and, ideally, attach examples of a good report and a bad report so that I can see the difference.

Looking at your repository, two things stand out:

You are using testInfo._steps that may break at any time, since it is not a part of Playwright API. I'd recommend writing a custom reporter instead and make use of the officially supported API that has all the step details.
You are calling test.afterEach() in a shared helper file. This does not work as you'd expect. See this comment for a thorough explanation and possible solutions.

symon-skelly1991 commented 7 months ago

Hi @dgozman, firstly, thanks for responding.

So, the junit-report rightly does contain test_result_steps, this was to help identify when the issue occurred.

on some tests that are run, the test_result_steps all pass, however the test did actually fail, and we can see this in the accompanying data inside of the test case in the junit-report.

I will do as you say and create a custom reporter and stop using the test.afterEach inside of my helper file, as I know this to be the cause of the issue already :)

microsoft / playwright