[Feature] TRX Reporter - Githubissues

creage commented 2 years ago

We are using Azure DevOps for our CI/CD, and report our test results using JUnit reporter for now. But this reporter is very limited in functionality - it cannot deliver attachments directly as a part of the report itself.

Would be nice if you guys there at Microsoft create a Visual Studio TRX reporter for Playwright, which has greater support in Azure, so we can deliver attachments (traces, screenshots, logs) directly to our pipeline summary page.

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=trx%2Cyaml#attachments-support

bigredjoe commented 2 years ago

I second this feature. We use Azure DevOps and a TRX reporter with attachments for playwright would really help us debug our e2e tests in our pipeline.

jwalakom commented 2 years ago

@creage @bigredjoe

We too want to generate videos as attachments for the failed functional test cases on our ADO Pipelines using Playwright. Can you please let me know if there is any other way to do this? Thanks!

creage commented 2 years ago

@jwalakom well, you can zip your attachments and download them using PowerShell task in your pipeline, and then publish them wherever you want. You can't attach your logs per test result, but you'll have them all available at least.

Camo30 commented 2 years ago

Any news about this issue? I would highly appreciate a TRX reporter for Playwright.

murraymichal commented 2 years ago

@jwalakom well, you can zip your attachments and download them using PowerShell task in your pipeline, and then publish them wherever you want. You can't attach your logs per test result, but you'll have them all available at least.

that's not what we are after. we'd like attachments to be available right in Tests section, as they should.

jwalakom commented 2 years ago

@jwalakom well, you can zip your attachments and download them using PowerShell task in your pipeline, and then publish them wherever you want. You can't attach your logs per test result, but you'll have them all available at least.

We have authored the tests in Azure Test plans with Test IDs. Then we "link" them to our functional tests. Playwright has a feature to generate test videos for failed test functional cases. Now we want to only attach these results ( videos in this case ) to our ADO pipeline as attachments.

Any thoughts on this? Thanks! @murraymichal @Camo30 @creage

ivan-xue-prospa commented 2 years ago

It will be awesone if Playwright has trx reporter built.

We were using Cypress for E2E testings and the name of screenshots/videos are based on the spec/test name, so we are attaching screenshots/videos using Powershell script. But for Playwright, the name of screenshots/videos is not that meaningful and it is hard to attach them using Powershell script.

If we can change/configure the name of screenshots/videos, it will be one workaround for this issue.

GP-QA commented 2 years ago

This would really help to quickly analyse failing tests within the test results of our Azure DevOps pipelines! Will this be picked up for next version 1.24?

OrangeTimes commented 2 years ago

This is much needed for Azure Devops users. None of the currently supported reporters doesn't give the ability to publish test results e.g. screenshots, video in Azure Devops

ShuiRuTian commented 2 years ago

I have created one naive TRX reporter: https://github.com/ShuiRuTian/trx-reporter-for-playwright

I did not concern much about some edge conditions for now(like what about the attachment only with content, where should we put them in TRX file?), but it starts working on Azure DevOps: (I use azure pipeline for the repo itself)

When it's matured enough, I would try to move it to the official repo(here!) if they accept it :)

ShuiRuTian commented 2 years ago

And if possible I want to get some feedback about

how you want to set some variables(like "owner" and "priority"), is the way in the readme good enough or not.
what do you want to control by configurations.

Let's bring a TRX reporter for playwright together.

I will keep an eye on both here and the repo, thanks guys!

mxschmitt commented 2 years ago

I have created one naive TRX reporter: https://github.com/ShuiRuTian/trx-reporter-for-playwright

I did not concern much about some edge conditions for now(like what about the attachment only with content, where should we put them in TRX file?), but it starts working on Azure DevOps: (I use azure pipeline for the repo itself)

When it's matured enough, I would try to move it to the official repo(here!) if they accept it :)

There is already an open PR, do you mind reviewing/comparing/collaborating on it? https://github.com/microsoft/playwright/pull/15736

ShuiRuTian commented 2 years ago

@mxschmitt

Oops, sorry I totally miss it.

It's kind of subtle, although I create this repo indeed for Azure, but in theory it's for TRX. Azure could use TRX to give a friendly UI and others could too (I have tested the trx generated by the reported could be opened by VS).

I will take a look at this PR to see the difference.

ShuiRuTian commented 2 years ago

@alexneo2003 friendly ping~

It's hard for me to know what really happens for azure reporter, I rarely use the Azure test plan and API directly. So the comparation is unfair, please correct me if I make any mistakes.

Azure-Reporter:

good:

flexible, better control with azure. Maybe powerful for advanced situation. As long as Azure has the feature, you could use it.
accurate. There is no mismatch, no more and no less, everything is from Azure and for Azure.

not good:

not out-of-box. Complex configutation, you need to set azure configuration totally by youself(even token, it's fine, but kind of strange).
Potentially lots of requests sent to Azure, might be a problem. Each case will send some requests when it ends. Will this influence the running of tests?
Tight couple with Azure.
Not able to set owner or priority for now, but it's easy to add them.

TRX-reporter:

good:

The TRX report might be consumed by others.
out-of-box. It's just like other reporters. You could see the report locally and easy to debug. Uploading result to cloud(like Azure) is another step.

not good:

it's just TRX, in the future if azure has a new feature and azure does not adapt it for trx or TRX could not support it, there is nothing we could do.

alexneo2003 commented 2 years ago

@alexneo2003 friendly ping~

It's hard for me to know what really happens for azure reporter, I rarely use the Azure test plan and API directly. So the comparation is unfair, please correct me if I make any mistakes.

Azure-Reporter:

good:

flexible, better control with azure. Maybe powerful for advanced situation. As long as Azure has the feature, you could use it.

accurate. There is no mismatch, no more and no less, everything is from Azure and for Azure.

not good:

not out-of-box. Complex configutation, you need to set azure configuration totally by youself(even token, it's fine, but kind of strange).

Potentially lots of requests sent to Azure, might be a problem. Each case will send some requests when it ends. Will this influence the running of tests?

Tight couple with Azure.

Not able to set owner or priority for now, but it's easy to add them.

TRX-reporter:

good:

The TRX report might be consumed by others.

out-of-box. It's just like other reporters. You could see the report locally and easy to debug. Uploading result to cloud(like Azure) is another step.

not good:

it's just TRX, in the future if azure has a new feature and azure does not adapt it for trx or TRX could not support it, there is nothing we could do.

I don't known what is TRX and not working with it.
About "lots of requests" read this
at my reporter using usual Azure DevOps API to publish results inside Azure DevOps Test Runs because you need to set config with some data including token.
- i don't known about what "owner" you wright
- if TRX reporter is best way for report test results - welcome. I'm publishing results via Azure DevOps API and don't known any other ways.

alexneo2003 commented 2 years ago

@alexneo2003 friendly ping~

It's hard for me to know what really happens for azure reporter, I rarely use the Azure test plan and API directly. So the comparation is unfair, please correct me if I make any mistakes.

Azure-Reporter:

good:

flexible, better control with azure. Maybe powerful for advanced situation. As long as Azure has the feature, you could use it.

accurate. There is no mismatch, no more and no less, everything is from Azure and for Azure.

not good:

not out-of-box. Complex configutation, you need to set azure configuration totally by youself(even token, it's fine, but kind of strange).

Potentially lots of requests sent to Azure, might be a problem. Each case will send some requests when it ends. Will this influence the running of tests?

Tight couple with Azure.

Not able to set owner or priority for now, but it's easy to add them.

TRX-reporter:

good:

The TRX report might be consumed by others.

out-of-box. It's just like other reporters. You could see the report locally and easy to debug. Uploading result to cloud(like Azure) is another step.

not good:

it's just TRX, in the future if azure has a new feature and azure does not adapt it for trx or TRX could not support it, there is nothing we could do.

Ok, as i seen TRX is native Azure DevOps file format which stored along source code and with help pipeline task can publish test results. It's most useful when tests running at Azure pipeline. When I'm started working with my reporter I'm not known about trx) and my reporter is most useful at any place running

ShuiRuTian commented 2 years ago

@alexneo2003 Thanks a lot for the sharing.

Technically, TRX is a file format and usually it's generated by VS test for C#. Here is an example. And just FYI, VS could open the TRX file, no surprising.

if TRX reporter is best way for report test results - welcome. I'm publishing results via Azure DevOps API and don't known any other ways.

It's hard to say TRX reporter is the best way, really. TRX is limited by its format, there are some situations that might be not easy to support, like if a playwright attachment only contains content rather than path, Azure report could handle this situation pretty well, but TRX only supports file path or url, it does not support content directly.

In my opinion, Azure-reporter might be much more powerful and flexible when user decides to put effort on Azure. And trx-reporter is an easy out-of-box option, only need to change reporter and few lines of azure pipeline config to something like this:

i don't known about what "owner" you wright

You could tell Azure the owner of the test case.

And similarly, you could set "priority", Azure could use it too.

Please remind/correct me if I miss/misunderstand anything

mxschmitt commented 2 years ago

Hi folks,

sorry for the late response, I was out of office. I was talking with the team, in general the TRX reporter and Azure DevOps reporter can coexist, if it's necessary for Playwright Test.

TRX is limited by its format, I was experimenting a while ago with it, especially when its about attachments. Also if you have a TRX file inside GitHub Actions, since you run your tests most likely there, then its hard or harder to upload it to Azure DevOps. Maybe this is a use-case nobody wants tho. The Azure DevOps reporter would just work fine, since it calls the Azure DevOps APIs directly.

@ShuiRuTian would the solution @alexneo2003 is working for you work for your use-case? Are you using Azure DevOps? (owners and priority is definitely something which should be added as well). Would you prefer opening the TRX file inside VS instead of using Playwright's HTML reporter? (you can open traces there etc. out of the box, so it's much more powerful).

ShuiRuTian commented 2 years ago

@mxschmitt Hi.

would the solution @alexneo2003 is working for you work for your use-case?

It would. Although I did not have a try, I am confident that it could meet all my requirements. There is nothing it can't do since it sends requests to Azure directly.

Are you using Azure DevOps?

Yes! The test pipeline is pretty friendly to open source project. Although the projects of my work are also on Azure DevOps, there are experts to maintain them.

Would you prefer opening the TRX file inside VS instead of using Playwright's HTML reporter?

Nop, I mentioned it just to prove the generated TRX file could be validated by visual studio. HTML reporter is absolutely better to view the results.

AndrewCraswell commented 1 year ago

I have created one naive TRX reporter: https://github.com/ShuiRuTian/trx-reporter-for-playwright

I did not concern much about some edge conditions for now(like what about the attachment only with content, where should we put them in TRX file?), but it starts working on Azure DevOps: (I use azure pipeline for the repo itself)

When it's matured enough, I would try to move it to the official repo(here!) if they accept it :)

I think this is the way forward. We've just started using this reporter and it's great! I would love to see this continued to be built out, then integrated directly into Playwright as a reporter. Microsoft maintains Playwright so first-party support for VSTest (TRX) format would enable seamless integration into Azure DevOps (and any other tooling that supports VS Test).

The alternative option to have a reporter execute Azure DevOps API requests was not ideal since it requires a token, which would require a service principal. And the requests would be made on each execution, which is not ideal. We'd like to generate the TRX file then decide which systems to ingest it to. That also gives us a pathway to integrate attachments into Github Actions, etc.

dpaquette commented 1 year ago

Is attachments being displayed in the Test Results panel the main thing folks are looking for here? If Azure DevOps were able to display attachments that are included in the existing junit reporter, would that meet everyone's needs?

AndrewCraswell commented 1 year ago

Correct me if I'm wrong, but I don't believe JUnit as a format supports attachments. TRX natively supports them.

dpaquette commented 1 year ago

There's not official support but there are a couple common ways that junit reports include attachments. The Playwright junit reported uses the approach of including them inside the <system.out> element.

<system-out>
<![CDATA[
[[ATTACHMENT|../test-results/LandingPage.test.tsx-my-test-suite-example1-chromium/video.webm]]

[[ATTACHMENT|../test-results/LandingPage.test.tsx-my-test-suite-example1-chromium/test-finished-1.png]]
]]>
</system-out>

There's a good write-up about this here: https://github.com/testmoapp/junitxml#attachments-in-test-output

Shanmuktej commented 1 year ago

The issue below has to be developed to improve the Azure workflow. [Feature] Enhanced Playwright Integration in Azure Pipelines Test Results #26682

woeterman94 commented 11 months ago

Any update on this?

dpaquette commented 11 months ago

We've made some improvements to the PublishTestResults task in Azure DevOps. It will now publish attachments for JUnit report files from Playwright. This currently works for Windows build agents and support is coming soon for other build agents.

dpaquette commented 9 months ago

JUnit attachment support is now officially available for all pipeline agents. With Azure Pipelines, you can now consume the built-int JUnit reports from Playwright and have attachments appear in test results. I suspect that most people won't need a trx reporter given the improved support for JUnit files within Azure Pipelines.

Check out the announcement blog post here: https://devblogs.microsoft.com/devops/junit-attachments-support-for-publish-test-results/

BryceBarbara commented 6 months ago

@dpaquette Do you know where people should go if there are bugs/issues with the publishing of attachments via JUnit reports?

dpaquette commented 6 months ago

@dpaquette Do you know where people should go if there are bugs/issues with the publishing of attachments via JUnit reports?

Report a bug here: https://developercommunity.visualstudio.com/AzureDevOps and then maybe link to it in this comment thread. I'll take a look at it once you've filed a bug.

BryceBarbara commented 6 months ago

@dpaquette Created a bug report for ya! https://developercommunity.visualstudio.com/t/PublishTestResults2-no-longer-uploads-J/10652153 I tried my best to fill out a decent bug report

dpaquette commented 6 months ago

@BryceBarbara Thanks! I will take a look at it.

BryceBarbara commented 6 months ago

My support post was closed and marked as a duplicate so there must be some known issue that isn't publicly known right now. If anyone is having issues with JUNIT attachments in Azure Devops, just know it's not a you thing.

dpaquette commented 6 months ago

My support post was closed and marked as a duplicate so there must be some known issue that isn't publicly known right now. If anyone is having issues with JUNIT attachments in Azure Devops, just know it's not a you thing.

It is definitely not only you. We've received other reports too. Still investigating. I replied to your comment on the other bug report. Greatly appreciate it if you can share debug logs with us privately over there.

microsoft / playwright

[Feature] TRX Reporter #11143