search

microsoft / playwright-testing-service

MIT License
70 stars 10 forks source link

Playwright Testing Service Dotnet support #86

Closed JanakiramanPachaiyappan closed 4 months ago

JanakiramanPachaiyappan commented 6 months ago

Most of our tests are written in dotnet. Do you have a plan to support dotnet playwright tests in the future? We would want to make use of the playwright testing service to run our tests (3000+) at scale. do you have any suggestions?

AB#1925497

vvs11 commented 6 months ago

@JanakiramanPachaiyappan , Please refer to this issue https://github.com/microsoft/playwright-testing-service/issues/16 for more details. We have experimental support for .NET.

JanakiramanPachaiyappan commented 6 months ago

@vvs11 I tried the following the example for .net, the tests are running in local instead of cloud browsers. I am using playwright version 1.0.40 and i have set the token and region url in the environment variables. Am i missing something?

vvs11 commented 6 months ago

That should do it. Are you using MSTest or NUnit test runner? Also, please check the environment variable name. It should be 'PLAYWRIGHT_SERVICE_ACCESS_TOKEN' and that it is accessible from the directory where you are running tests from. If you are seeing any errors, please share that too.

JanakiramanPachaiyappan commented 6 months ago

@vvs11 - I am using NUnit test runner. Environment variables are also set correctly and is accessible from the directory where I am running the tests from. There are no errors, but the tests are running in local instead of cloud browsers. Do we need any config file similar to playwright.service.config.ts?

JanakiramanPachaiyappan commented 6 months ago

@vvs11 update: I cloned the nunit tests from the sample repo and that works. the tests are running in cloud browser. The problem is when I try to run tests from my repo it runs in local. Note: In my repo, the tests are written in bdd using specflow. does that have anything to do with this issue?

vvs11 commented 6 months ago

specflow should not be an issue. Could you try setting the environment variable 'DEBUG' with value 'pw*' and try? You can collect the logs in a file and share it with us to look at.

vvs11 commented 6 months ago

@JanakiramanPachaiyappan Just checking if you were able to get it to work? Please reach out to us at playwrighttesting@microsoft.com to share the logs. We can set up some time to debug this together.

maitraballari commented 6 months ago

I am seeing the exact same issue when I run the tests using NUnit. All my tests seem to be running locally. All I did was setup a new workspace in the MS Playwright Testing Service, I defined the 2 environment variables (PLAYWRIGHT_SERVICE_ACCESS_TOKEN, PLAYWRIGHT_SERVICE_URL) in the .env file. I do not see the test runs in cloud at all. Wondering what am I missing here. Do we need to add anything else?

puagarwa commented 6 months ago

I am seeing the exact same issue when I run the tests using NUnit. All my tests seem to be running locally. All I did was setup a new workspace in the MS Playwright Testing Service, I defined the 2 environment variables (PLAYWRIGHT_SERVICE_ACCESS_TOKEN, PLAYWRIGHT_SERVICE_URL) in the .env file. I do not see the test runs in cloud at all. Wondering what am I missing here. Do we need to add anything else?

HI @maitraballari can you please confirm your playwright version? and please set the environment variable 'DEBUG' with value 'pw*' and share the logs so that we can take a look. And also mention your subscriptionId and workspace Name/Id. Since logs can contain sensitive info please share it over email playwrighttesting@microsoft.com

maitraballari commented 6 months ago

Sent an email @puagarwa . Thanks!

JanakiramanPachaiyappan commented 5 months ago

specflow should not be an issue. Could you try setting the environment variable 'DEBUG' with value 'pw*' and try? You can collect the logs in a file and share it with us to look at.

@vvs11 - Specflow has its own Hooks for BeforeScenario/BeforeFeature/BeforeTestRun. But the environment variables for playwright testing service is set at the time of registering the browserservice using the Nunit [Setup] attribute. How does the playwright testing service environment variables get set with Specflow as nunit [setup] hook is never called? Please clarify. It would be great if you could post some examples with specflow integration.

puagarwa commented 5 months ago

Sent an email @puagarwa . Thanks!

HI @maitraballari I checked our telemetry and we dont see any request from your subscription so far. Would you mind trying out our sample Nunit project first https://github.com/microsoft/playwright-testing-service/tree/main/samples/.NET/NUnit to make sure you can run this successfully.

As you mentioned logs are large, you could change the variable to 'DEBUG' with value 'pw:browser*' to reduce log size.

puagarwa commented 5 months ago

specflow should not be an issue. Could you try setting the environment variable 'DEBUG' with value 'pw*' and try? You can collect the logs in a file and share it with us to look at.

@vvs11 - Specflow has its own Hooks for BeforeScenario/BeforeFeature/BeforeTestRun. But the environment variables for playwright testing service is set at the time of registering the browserservice using the Nunit [Setup] attribute. How does the playwright testing service environment variables get set with Specflow as nunit [setup] hook is never called? Please clarify. It would be great if you could post some examples with specflow integration.

HI @JanakiramanPachaiyappan You are right here about how service is integrated with NUnit via browserService class and if in your case its not called then yes service integration wont work out of the box. But we could make it work by changing in how you launch your browser by replacing it with logic similar to this. We could help you provide sample fixture if you could share small repro for your test suite.

puagarwa commented 5 months ago

Sent an email @puagarwa . Thanks!

HI @maitraballari I checked our telemetry and we dont see any request from your subscription so far. Would you mind trying out our sample Nunit project first https://github.com/microsoft/playwright-testing-service/tree/main/samples/.NET/NUnit to make sure you can run this successfully.

As you mentioned logs are large, you could change the variable to 'DEBUG' with value 'pw:browser*' to reduce log size.

HI @maitraballari I ack your mail you sent with pw:browser logs and it confirm that your test suite is launching local browser due to manual browser launch that you have in your code. please see below response

I did run the sample NUnit tests from your git repo and I can see one test run showing up in the MS Playwright testing service Dashboard. However I cannot view any logs or even test results from here. Not sure why.

This is intended behavior as currently we only offer activity log without any test artifacts, etc. Reporting feature is in our backlog.

I am guessing the issue I have with my tests is that I have a setup step in each of my Nunit tests which launches a browser. Guessing that the BrowserService needs to be integrated here so that we can use the Playwright testing service while creating the browser. Can you confirm if this could be the case? And if I should be setting up the browser service somewhere in my code.

Yes, this is case. We need to integrate service by replacing local browser launch with service browser launch in your setup file. In detail, we have to remove browserType.LaunchAsync with browserType.ConnectAsync. Here is example on how it can be done.

maitraballari commented 5 months ago

@puagarwa, thank you for all your help and suggestions. Following your advice I can now run all the tests from the Azure Devops pipeline in Playwright Testing Service and view the test results directly from the pipeline.Appreciate your help and time!

puagarwa commented 5 months ago

@puagarwa, thank you for all your help and suggestions. Following your advice I can now run all the tests from the Azure Devops pipeline in Playwright Testing Service and view the test results directly from the pipeline.Appreciate your help and time!

Glad you are able to integrate service. feel free to open issue if you face any issue or need our help.

puagarwa commented 4 months ago

gistering the browserservice using the Nunit [Setup] attribute. How does

@JanakiramanPachaiyappan Just checking if you are unblocked in your dotnet integration with service, let me know if you are facing issue.

JanakiramanPachaiyappan commented 4 months ago

gistering the browserservice using the Nunit [Setup] attribute. How does

@JanakiramanPachaiyappan Just checking if you are unblocked in your dotnet integration with service, let me know if you are facing issue.

@puagarwa - We can close this issue. dotnet integration with the service works fine for me. Thanks!

uday2029 commented 1 month ago

Hi, I'm able to integrate the service, and i'm able to run in the cloud as well. Thank you @puagarwa for the help, but i'm facing an issue, which is, i'm not able to run with multiple workers, i'm using NUnit test runner. I have configured in .runsettings, but it doesn't work for me. Everytime i ran with multiple testworkers, it is showing only 1 worker in the cloud. Can you please help me on this.