search

microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
66.41k stars 3.63k forks source link

[Feature]: Show the Test Annotation in the Merged Report Even Though the Test are Being Skipped #30260

Closed aldys33 closed 6 months ago

aldys33 commented 6 months ago

🚀 Feature Request

I just notice that if my test got skipped, the annotation is missing from the report. This only happen when I use merge-report, which is I'm utilizing sharding. image

I think it's better to have annotation still got attached in the report, despite the test being success, failed, or even skipped as well for the merged report.

Example

Condition when the test got skipped

image

Condition when the test is failed/success

image

Motivation

I believe there are user that utilize annotation for several purpose. Like for me, it is a place to store Test ID that connects with the AIO. So it's important to preserve the annoation in the report regardless of how the test execution is going

dgozman commented 6 months ago

@aldys33 Could you please share a repro exhibiting the issue that we can run locally?

aldys33 commented 6 months ago

yeah, will create a repo to demonstrate on that one

aldys33 commented 6 months ago

@dgozman Please refer to the following repo https://github.com/aldys33/Playwright.Annotation.Demo/actions I did create a sample run with and without sharding. Please note that when using sharding, the annotation is not showing

dgozman commented 6 months ago

@aldys33 Thank you for the repro, I can see the bug now.

Investigation notes: a test is skipped in one of the shards and there is no onTestEnd event in the blob reporter that contains annotations.

aldys33 commented 6 months ago

Thanks, @dgozman