[FEATURE REQ] Show report button on the Activity Log page

[elvisduric]

Is your feature request related to a problem? Please describe. With this playwright service, we could run the test cases but the test report will still be on our local machine (or stored as a GitHub artefact, etc.). It would be helpful and beneficial to have the option to present test run reports on the Playwright Portal page.

Describe the solution you'd like Currently, on the Playwright Portal, we can see each test run we started over the Playwright service. The small button or another indication for the report could be helpful and beneficial to the user to immediately see the results of that run.

AB#1890161

[dmcweeney]

+1

Use case - to have the run and traces immediately available for the team members. To be able to view the run logs etc from the portal is a big + as downloading the logs from the AzDO/Github artifacts and expanding the zip is a pain.

[SebaRossetti]

+1

[vvs11]

Thanks for raising this. We are collecting feedback for this request. Please 👍 on the issue if you want us to build this. Also, share the use-cases that you want to enable through this in the comments.

[chaitanya-cloudknox]

For us we run the tests everyday on production in different regions/environments and it would be a central place for an oncall engineer or anyone going through the failures.

[dfibuch]

This confused me a bit, I run bunch of tests but couldn't actually see if they were passing or failing, just tells me how long they took and how many run. I assumed that there would be some sort of reporting to see easily what the result is and then potentially be able to share it with others so they can see them.

Going even further, it would be great if its possible to extract the result data or have some API that can be called and build a status page so you can easily see if anything is failing. Use case here is that we would run all the regression tests and easily identify if things are suddenly not working that have not been directly affected by the latest development.

Side note: I tried to follow this article (https://learn.microsoft.com/en-us/azure/playwright-testing/monitor-playwright-testing) to try and get some sort of logging and I've hooked up both Log Analytics and Storage Account in the Diagnostic Settings of my Playwright service, but I am getting no data. Am I doing something wrong or is that not how it's supposed to be used?

[ranat5]

+1 Use case - We would like to use Playwright testing as service offer. Currently, when we try to run the tests, we do see the run result under the Activity Log but displaying the results of the tests in the dashboard will help us immensely to visualise the test results (something similar to the process that we can visualise on our browsers locally when we run: npx playwright show-report)

[RazvanFilipescu]

+1

My use case would be having a comprehensive reporting tool embedded in the service. Some of the features that would make it great are:

• Having pre-defined and customizable graphical reports (pie charts, bar graphs, etc).
• Displaying test results, per test run. This should be in a graphical way (color-coded e.g. with red for failed and green for passed) as well as having a detailed log of the traces currently being displayed locally.
• Displaying summary test results, for a number of test runs (e.g. maybe show all test runs and their results per time interval)
• Having the option of saving the Playwright *.zip trace files in an Azure Container e.g. and having these readily available beside the test result if you want to investigate further.
• Highlighting flaky tests per test run (if you have a retry mechanism enabled) and per a selection of test runs (where one test may fail in one run but be ok in another).
• (possibly unrelated but might make use of the historic test run information - reports) Having the option to orchestrate/optimize test runs based on previous test run data. An example would be starting the longest duration test on the first machine, the second longest on the second machine, etc. and, once all machines are used up, start the pending longest test on the first machine that becomes available.

[magesh-chandran]

+1 To start with,

1. would like to have link to test results with meta data like release, browser, OS, date, ...
2. Would like to have ability to Create my own reports (charts, Pie, table) using meta data as filters .. maybe something like powerBI

[chris-strudel]

+1 I'd take it even further. I think the videos and traces should be available in the ADO pipeline run itself. The integration should be seamless. That's currently the major limitation of ADO in my opinion - test reporting. Each test case run should have it's associated video and trace available for immediate viewing in ADO.

The documentation has the build step:

• task: PublishPipelineArtifact@1 displayName: Upload Playwright report inputs: targetPath: '$(MainProjectDirectory)/test-results' # update accordingly artifact: 'Playwright tests' publishLocation: 'pipeline'

If we're going to publish to the pipeline then publish to the pipeline all the way!

[puagarwa]

@chris-strudel There is issue in ADO pipelines from quite some time with thier JUNIT file parsing which cause artifacts to not show up in ADO test tab. I myself have filed this issue in thier repo sometime but unfortunately there is no traction. https://github.com/microsoft/azure-pipelines-tasks/issues/16349

[PagelsR]

+1 playwright test dashboard in the portal.

[puagarwa]

+1 I'd take it even further. I think the videos and traces should be available in the ADO pipeline run itself. The integration should be seamless. That's currently the major limitation of ADO in my opinion - test reporting. Each test case run should have it's associated video and trace available for immediate viewing in ADO.

The documentation has the build step:

• task: PublishPipelineArtifact@1 displayName: Upload Playwright report inputs: targetPath: '$(MainProjectDirectory)/test-results' # update accordingly artifact: 'Playwright tests' publishLocation: 'pipeline'

If we're going to publish to the pipeline then publish to the pipeline all the way!

@elvisduric and @chris-strudel ADO team recently anounced the attachment support for JUNIT, We should get playwright artifacts with every test now in ADO https://devblogs.microsoft.com/devops/junit-attachments-support-for-publish-test-results/

[chris-strudel]

+1 I'd take it even further. I think the videos and traces should be available in the ADO pipeline run itself. The integration should be seamless. That's currently the major limitation of ADO in my opinion - test reporting. Each test case run should have it's associated video and trace available for immediate viewing in ADO. The documentation has the build step:

• task: PublishPipelineArtifact@1 displayName: Upload Playwright report inputs: targetPath: '$(MainProjectDirectory)/test-results' # update accordingly artifact: 'Playwright tests' publishLocation: 'pipeline'

If we're going to publish to the pipeline then publish to the pipeline all the way!

@elvisduric and @chris-strudel ADO team recently anounced the attachment support for JUNIT, We should get playwright artifacts with every test now in ADO https://devblogs.microsoft.com/devops/junit-attachments-support-for-publish-test-results/

Hi @puagarwa! Thank you for letting us know. I followed the link and set it up. That is a step in the right direction, but what we really need is to be able to view those traces and videos within the ADO tests tab. There needs to be some UX support. When you have 100+ tests and multiple failures, it just isn't practical to have to download each video one at a time and then launch it in a viewer. Also, at present, I don't think there is a way to consume the traces that are created. Those are usually shown in the HTML reporter. When support for those features is added, I think Playwright will surge forward in adoptability. As it stands right now, the reporting is still the weakest spot for Playwright with ADO.

[puagarwa]

+1 I'd take it even further. I think the videos and traces should be available in the ADO pipeline run itself. The integration should be seamless. That's currently the major limitation of ADO in my opinion - test reporting. Each test case run should have it's associated video and trace available for immediate viewing in ADO. The documentation has the build step:

• task: PublishPipelineArtifact@1 displayName: Upload Playwright report inputs: targetPath: '$(MainProjectDirectory)/test-results' # update accordingly artifact: 'Playwright tests' publishLocation: 'pipeline'

If we're going to publish to the pipeline then publish to the pipeline all the way!

@elvisduric and @chris-strudel ADO team recently anounced the attachment support for JUNIT, We should get playwright artifacts with every test now in ADO https://devblogs.microsoft.com/devops/junit-attachments-support-for-publish-test-results/

Hi @puagarwa! Thank you for letting us know. I followed the link and set it up. That is a step in the right direction, but what we really need is to be able to view those traces and videos within the ADO tests tab. There needs to be some UX support. When you have 100+ tests and multiple failures, it just isn't practical to have to download each video one at a time and then launch it in a viewer. Also, at present, I don't think there is a way to consume the traces that are created. Those are usually shown in the HTML reporter. When support for those features is added, I think Playwright will surge forward in adoptability. As it stands right now, the reporting is still the weakest spot for Playwright with ADO.

@chris-strudel Yes, we definitely see the reason to provide dedicated playwright reporting hence we are actively working on it.

[chris-strudel]

+1 I'd take it even further. I think the videos and traces should be available in the ADO pipeline run itself. The integration should be seamless. That's currently the major limitation of ADO in my opinion - test reporting. Each test case run should have it's associated video and trace available for immediate viewing in ADO. The documentation has the build step:

• task: PublishPipelineArtifact@1 displayName: Upload Playwright report inputs: targetPath: '$(MainProjectDirectory)/test-results' # update accordingly artifact: 'Playwright tests' publishLocation: 'pipeline'

If we're going to publish to the pipeline then publish to the pipeline all the way!

@elvisduric and @chris-strudel ADO team recently anounced the attachment support for JUNIT, We should get playwright artifacts with every test now in ADO https://devblogs.microsoft.com/devops/junit-attachments-support-for-publish-test-results/

Hi @puagarwa! Thank you for letting us know. I followed the link and set it up. That is a step in the right direction, but what we really need is to be able to view those traces and videos within the ADO tests tab. There needs to be some UX support. When you have 100+ tests and multiple failures, it just isn't practical to have to download each video one at a time and then launch it in a viewer. Also, at present, I don't think there is a way to consume the traces that are created. Those are usually shown in the HTML reporter. When support for those features is added, I think Playwright will surge forward in adoptability. As it stands right now, the reporting is still the weakest spot for Playwright with ADO.

@chris-strudel Yes, we definitely see the reason to provide dedicated playwright reporting hence we are actively working on it.

If I can be of any assistance please let me know. I think Playwright is the most innovative test automation tool I've seen in the last fifteen years. I'm super excited for the enhanced reporting! That is seriously the only thing holding us back from going full throttle with the Playwright Testing Service and ADO.

[vvs11]

Hi Folks,

We are excited to introduce the new reporting feature for the Microsoft Playwright Testing service. With this feature, you can publish your test results and artifacts to the service and view them in the MPT portal. You'll have all the information needed to diagnose failed tests. The portal also hosts all collected artifacts, including the trace viewer. Integrating this feature with your existing test suite is straightforward and easy.

You can read more about it here: https://aka.ms/mpt/reporting-announce

Currently, the feature is invite-only preview. You can sign up using this link: https://aka.ms/mpt/reporting-signup

Thanks