[Feature] Playwright integration with Azure DevOps Test Plans

[PavanMudigondaTR]

Is it possible to provide support for updating test results in Azure DevOps Test Plans by using Playwright directly ? I have come across this feature that is provided in Webdriver.IO. The test case execution status for Automated test cases are currently having to be updated manually as either passed or failed. And also the Automation status drop down we are not able to update it as Automated.

https://webdriver.io/docs/gmangiapelo-wdio-azure-devops-service https://github.com/gianlucamangiapelo/wdio-azure-devops-service/tree/main/.github/workflows

[dgozman]

Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.

[PavanMudigondaTR]

Thanks a lot for quick reply! I will give this a try!

[PavanMudigondaTR]

Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.

@dgozman It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?

[dgozman]

It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?

Well, this is actually an integration between the test runner and ADO. @playwright/test for JS/TS includes a test runner, and this reporter integrates with ADO.

For python we recommend pytest, so you will need pytest+ADO integration, perhaps something like this?

Similarly, for Java you would need something that brings together JUnit+ADO, and for C# something that supports MSTest or NUnit with ADO.

[PavanMudigondaTR]

It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?

Well, this is actually an integration between the test runner and ADO. @playwright/test for JS/TS includes a test runner, and this reporter integrates with ADO.

For python we recommend pytest, so you will need pytest+ADO integration, perhaps something like this?

Similarly, for Java you would need something that brings together JUnit+ADO, and for C# something that supports MSTest or NUnit with ADO.

@dgozman I get it now. Playwright Test is only Js/TS Runner. I understand. So we need similar solutions for Pytest, NUnit and Junit5 as well now. I would appreciate if this can be looked at playwright team or some community members who can ? I see Azure DevOps has node api, python api, C# api and go api which can be used to build this integration. But i am not that skilled to do this. The pip library link you sent in previous reply will not solve it fully.

[daniavander]

how is it fill:

[daniavander]

I get this in the log azure: Test [31035] Smoke test - Activity list @response - passed azure: Start publishing: [31035] Smoke test - Activity list @response SUITE PASS :) azure: Start upload attachments for test case [31035] azure: Result published: [31035] Smoke test - Activity list @response azure: Run 1000284 - Completed

but i see nothing in test case

[PavanMudigondaTR]

I get this in the log

azure: Test [31035] Smoke test - Activity list @response - passed

azure: Start publishing: [31035] Smoke test - Activity list @response

SUITE PASS :)

azure: Start upload attachments for test case [31035]

azure: Result published: [31035] Smoke test - Activity list @response

azure: Run 1000284 - Completed

but i see nothing in test case

Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!

[daniavander]

o i see, so we are waiting it !

[mechandan]

@Dgozman Any official update on integration playwright typescript/javascript with the ADO test plan ?? Here is a third-party reporter it just take a single plan id ( suite id ) at a time , in every sprunt the suite id changes , ( much appreciate the effort from the developer n thankyou! )

[daniavander]

Is there any progression with playwright automated test results connect with azure devops manual test cases?

[joeflan]

Yeah I'd also

Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!

Is there any place to track this, or any idea on when this integration might happen?

[rowel14]

I get this in the log azure: Test [31035] Smoke test - Activity list @response - passed azure: Start publishing: [31035] Smoke test - Activity list @response SUITE PASS :) azure: Start upload attachments for test case [31035] azure: Result published: [31035] Smoke test - Activity list @response azure: Run 1000284 - Completed but i see nothing in test case

Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!

Do we have update on this one? I got the same result. My test case didn't update

[rowel14]

Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.

@dgozman It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?

could you share some screenshots on how you wer able to work it?

[PavanMudigondaTR]

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.

Please note if you are committing your repo to GitHub make sure use secrets

Instead of passing token directly use secrets.

INSTEAD OF DOING:

token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',

DO:

token: process.env.ADO_TOKEN

to run it locally

export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds

FINALLY in GitHub Actions you can do like this

pass in

• name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test

[rowel14]

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.

Please note if you are committing your repo to GitHub make sure use secrets

Instead of passing token directly use secrets.

INSTEAD OF DOING:

token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',

DO:

token: process.env.ADO_TOKEN

to run it locally

export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds

FINALLY in GitHub Actions you can do like this

pass in

• name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test

I'm actually using Azure DevOps, I'm still struggling to update the result of a specific test case in my test plan in Azure Test Plan,

[rowel14]

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.

Please note if you are committing your repo to GitHub make sure use secrets

Instead of passing token directly use secrets.

INSTEAD OF DOING:

token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',

DO:

token: process.env.ADO_TOKEN

to run it locally

export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds

FINALLY in GitHub Actions you can do like this

pass in

• name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test

I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.

[PavanMudigondaTR]

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only. Please note if you are committing your repo to GitHub make sure use secrets Instead of passing token directly use secrets. INSTEAD OF DOING: token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds', DO: token: process.env.ADO_TOKEN to run it locally export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds FINALLY in GitHub Actions you can do like this pass in

• name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test

I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.

Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman

I am using a tool called SpecSync currently due to few problems I faced.

[rowel14]

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only. Please note if you are committing your repo to GitHub make sure use secrets Instead of passing token directly use secrets. INSTEAD OF DOING: token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds', DO: token: process.env.ADO_TOKEN to run it locally export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds FINALLY in GitHub Actions you can do like this pass in

• name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test

I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.

Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman

I am using a tool called SpecSync currently due to few problems I faced.

Yeah, this is what I am talking about, I want to update the outcome (result) of each test cases in my test plan in Azure Test Plan. Do we have already a solution for this? Does Playwright will support this thing soon? Any alternative that we can use?

[Guru426]

@rowel @PavanMudigondaTR

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only. Please note if you are committing your repo to GitHub make sure use secrets Instead of passing token directly use secrets. INSTEAD OF DOING: token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds', DO: token: process.env.ADO_TOKEN to run it locally export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds FINALLY in GitHub Actions you can do like this pass in

• name: run playwright tests env: ADO_TOKEN: ${{ secrets.ADO_TOKEN }} shell: bash run: npx playwright test

I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.

Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman I am using a tool called SpecSync currently due to few problems I faced.

Yeah, this is what I am talking about, I want to update the outcome (result) of each test cases in my test plan in Azure Test Plan. Do we have already a solution for this? Does Playwright will support this thing soon? Any alternative that we can use?

Hello guys, Were you able to update the results to the test plans in Azure now ? If so, kindly share few inputs on how did you achieved it when you get sometime. Thanks!

[ravindrakumar6]

getting this error while results got published,

this is my test case automation status in TFS

[ravindrakumar6]

Can you please help me on this

[daniavander]

here is my log: azure: Uploading attachments for test: [31036] - Smoke test - Reports page @response @smok azure: Uploaded attachment azure: Left to publish: azure: Test results published for 14 test(s azure: Run 1000348 - Complete

where set the automation status and automation complexity? are these own fields no?

[ravindrakumar6]

@daniavander ,

Please find these screenshots hope this helps

TFS Automation Options available

MyPlaywright Config

Test Cases -- see the error down below

Same Test Case ID i used

See these details too

After analyzing all these still something is missing or not sure where exactly it's going wrong if you get any idea great help.

[ravindrakumar6]

@daniavander meanwhile please check this one provided by @alexneo2003

https://docs.specsolutions.eu/specsync/contact/troubleshooting#test-result-publishing-fails-with-mismatch-in-automation-status-of-test-case-and-test-run

so still trying to figure it out why this error is appearing. Please try to help if any ideas or hint that can resolve this issue Thanks.

[daniavander]

@ravindrakumar6 sorry, i do not know what is the problem. In ma devops i have only two status for automation status

in the config i do not add the suite id just the planid, perhaps

[daniavander]

@aslushnikov are there any future time, when would you like to take this integration to the official release?

[thomas-phillips-nz]

I tried asking questions of the playwright team in every livestream I could during Microsoft Build but it sounds like the Playwright team aren't familiar with Azure Test Plans. They kept referring to running tests in Azure Devops CI/Pipelines, but it doesn't sound like it will work with Azure Test Plans.

Would be good if you as a Microsoft team could support the Azure Test Plans feature.

[PavanMudigondaTR]

I tried asking questions of the playwright team in every livestream I could during Microsoft Build but it sounds like the Playwright team aren't familiar with Azure Test Plans. They kept referring to running tests in Azure Devops CI/Pipelines, but it doesn't sound like it will work with Azure Test Plans.

Would be good if you as a Microsoft team could support the Azure Test Plans feature.

There are two options currently. I recommend you try out in below sequence.

• OpenSource Solution using Third Party Report
• Paid Tool SpecSync

[Oscaruzzo]

This would be incredibly useful. I'm using the third-party reporter other people mentioned, at the moment https://github.com/alexneo2003/playwright-azure-reporter but a "built-in" integration would be awesome. Also integrated in the VSCode pluging, hopefully.

[nguyenquangtay]

upvote, thanks!

[HannahTran96]

Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.

Hi, could you support me, I always got No test points found for test case [4526] associated with test plan 4496. Check, maybe testPlanId, what you specified, is incorrect although my testPlanId and my testcase id is correct. Could you help me point out what part I should fix @dgozman []

[alexneo2003]

Hi, could you support me, I always got No test points found for test case [4526] associated with test plan 4496. Check, maybe testPlanId, what you specified, is incorrect although my testPlanId and my testcase id is correct. Could you help me point out what part I should fix @dgozman []

Read this , pls

[alexneo2003]

Hi, could you support me, I always got No test points found for test case [4526] associated with test plan 4496. Check, maybe testPlanId, what you specified, is incorrect although my testPlanId and my testcase id is correct. Could you help me point out what part I should fix @dgozman []

Read this , pls

[HannahTran96]

@

Hi, could you support me, I always got No test points found for test case [4526] associated with test plan 4496. Check, maybe testPlanId, what you specified, is incorrect although my testPlanId and my testcase id is correct. Could you help me point out what part I should fix @dgozman []

Read this , pls

Thank you, it works. I have question I want to combine playwright report and cucumber but I dont know how to do it, I tried some ways but it does not work. Could you help me @alexneo2003

[justinteix]

Hello, are there any updates on whether Microsoft is still looking to integrate Azure DevOps Test Plans into Playwright built-in? I'm new to Playwright and really would like to update my test case executions on my ADO test plans automatically during my CI runs. I've used other tools that have had this feature for a while and would really love to see it in Playwright.

[WPatryk]

Hello @aslushnikov , do you know if there is any plan to implement this anytime soon? Is there anything we could do to make it possible for near future? It looks like a lot of people would find this useful. It would be a great addition to PW integration with ADO.

[jonathanrossTPA]

I am hoping that Playwright recorder is also added to the Azure Devops Test & Feedback browser add-on, which would be a logical solution

On Thu, 23 May 2024 at 10:45, WPatryk @.***> wrote:

Hello @aslushnikov https://github.com/aslushnikov , do you know if there is any plan to implement this anytime soon? Is there anything we could do to make it possible for near future? It looks like a lot of people would find this useful.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/playwright/issues/14839#issuecomment-2126445446, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUJM4YC6THHZMAETGFVC2MDZDWNBVAVCNFSM5YVAFTMKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJSGY2DINJUGQ3A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

This email and any files transmitted with it contain confidential information and/or privileged or personal advice. This email is intended for the addressee(s) stated above only. If you are not the addressee of the email please do not copy or forward it or otherwise use it or any part of it in any form whatsoever. If you have received this email in error please notify the sender and remove the e-mail from your system. Thank you.

This is an email from the company Just Eat Takeaway.com N.V., a public limited liability company with corporate seat in Amsterdam, the Netherlands, and address at Piet Heinkade 61, 1019GM Amsterdam, registered with the Dutch Chamber of Commerce with number 08142836 and where the context requires, includes its subsidiaries and associated undertakings.

[brianpalma]

Is there any official update on integration playwright typescript/javascript with the ADO test plan?

[WPatryk]

Hi @dgozman is there any way we could push this forward to the development team? Do you know if there are any plans to put this issue into work?

[gcelia86]

Please can we have an update on this? Within QA requirement traceability is key, the ability to automatically associate a playwright test run with a test case (therefore a user story) would be great. As it stand the test that are executed by our ADO pipeline are completely separate to our test plans and main source of reporting.