Closed kiruba07 closed 6 months ago
dgozman
commented
6 months ago @kiruba07 Playwright follows a strict rule - one execution of npx playwright test equals a single report. If you would like multiple reports to be generated, run npx playwright test multiple times with different filters. Alternatively, do not use the html report as execution evidence, but something else - some json struct, or existing test attachments like a trace.
kiruba07
commented
6 months ago @kiruba07 Playwright follows a strict rule - one execution of
npx playwright testequals a single report. If you would like multiple reports to be generated, runnpx playwright testmultiple times with different filters. Alternatively, do not use the html report as execution evidence, but something else - some json struct, or existing test attachments like a trace.
@dgozman I would like to execute the amount of tests using the tags in a single trigger let's say npx playwright test --tags='my filters' and generate the report. I don't think so triggering the scripts multiple times with npx command is a viable solution as the triggers (single trigger for each tags) are running on the CI environment which is using shell interpreter commands and the scripts are filtered based on the tags.
I understand that I can use JSON struct for the evidence, but my business and PO's are more over interested to have a HTML view report that too with screenshots and traces enabled.
We have huge amount tests to run on every regression cycles and sign off is based on the evidences attached for the runs.
So basically we are stuck here to generate the individual html reports for each spec.
dgozman
commented
6 months ago @kiruba07 Thank you for the explanation. If you upload the full html report, and then attach deep links to a particular test in that report, would that work for you? Here is an example where we have the full html report, but can also link to a particular test.
kiruba07
commented
6 months ago @kiruba07 Thank you for the explanation. If you upload the full html report, and then attach deep links to a particular test in that report, would that work for you? Here is an example where we have the full html report, but can also link to a particular test.
@dgozman Thanks for your response. Yes, I am also getting the same type of full HTML report.
For Example: If execute 30 spec files in a single trigger and each of them are tagged with unique test id in JIRA. In this case full HTML will have the details for all 30 specs and it has deep link to each individual spec details.
The problem here is how to extract the deep link HTML that are specific to each spec and that can be used to upload as an evidence in to JIRA Xray for each individual test ID (spec file).
Also, I believe if I enable the traces that also will be part of the deep link. So I have to access the full html that contains 30 spec details and navigate to each deep linked spec file to view its corresponding details that includes traces as well
Would be great if there is a way to extract the deep linked spec details in the html format that contains all the execution details including traces (if trace was enabled).
Note: In JIRA XRAY , we can upload the execution attachements as .html file or .zip file that contains the dependencies to render the html. So we need download the attachment to view the evidences. In this case attaching full html for each test case id (spec file) will create confusion for the auditors who are reviewing the evidences.
Thanks, Kirubanandh G
dgozman
commented
6 months ago @kiruba07 I have looked closer, and it seems like you want a report per spec file, not per a single test? In any case, you can create a link to the html report with a pre-filtered test list. For example:
From this link, it's a single click to see the result, including traces.
The problem here is how to extract the deep link HTML that are specific to each spec and that can be used to upload as an evidence in to JIRA Xray for each individual test ID (spec file).
As you point out, extracting a link with a test ID is not that easy. The easiest way I can think of is to write a Playwright script to load the html report, click on each test and grab the url it navigates to.
In this case attaching full html for each test case id (spec file) will create confusion for the auditors who are reviewing the evidences.
Yeah, I'd recommend to upload the full html report only once, e.g. for the first spec file. And then link to it from all other specs.
dgozman
commented
6 months ago There seems to be no action item for Playwright here, so I'll close the issue. I don't think we'll generate multiple reports per run, so you'll have to work around that in some way - using deep links, running multiple times or something else.
Thank you for the issue! If you encounter more problems, don't hesitate to file new issues.
🚀 Feature Request
Generate individual HTML reports for each spec that is executed.
Problem Consider I am executing 30 tests (30 spec files and each spec file tagged with a unique JIRA XRAY Test case ID) and I have a mechanism to update the execution status (pass/fail, start time, and finish time ) automatically to XRAY (JIRA test management tool) after the execution.
Currently, I can parse the built-in Playwright JOSN report to capture the execution details for 30 spec files that were executed and map it to a unique JIRA ID tagged in the spec file, and use it for the integration with XRAY by using the XRAY Rest API's
Along with the execution status, I must upload the test execution evidence for each spec file (JIRA XRAY Test case ID). I am using built-in HTML reports as execution evidence. The problem is the HTML report contains the details for all the 30 spec files that were executed. Because of this, I cannot upload this HTML into JIRA XRAY as evidence for the particular JIRA XRAY Test case ID.
Ask:
It will be helpful if Playwright enables the flexibility to generate HTML reports for individual executed specs. So that we can use the generated individual HTML report for post-execution integrations, in this case, JIRA XRAY evidence upload.
Options
Something should be able to configure it on the playwright.config like below,
reporter: [['html', { outputFolder: 'my-report', individualHTML:true }]],Example
No response
Motivation
Provides more flexibility to use the built-in solution for the post-execution integration activities like updating the execution status on JRA XRAY after each execution.