Playwright/Browsers hang when having more than 3 (sometimes 2) instances running at the same time when testing a Blazor Server App

[dardanlibrary]

Context:

• Playwright Version: 1.23.0
• Operating System: Windows 11
• Node.js version: 16.13.0
• Browser: Chromium, Firefox

Code Snippet

using var playwright = await Playwright.CreateAsync();
await using var browser = await playwright.Firefox.LaunchAsync(
    new BrowserTypeLaunchOptions
    {
        Headless = false
    }
);

await Parallel.ForEachAsync(
    Enumerable.Range(1, options.Iterations),
    new ParallelOptions { MaxDegreeOfParallelism = options.Iterations },
    async (iteration, cancellationToken) =>
    {
        await using var browserContext = await browser.NewContextAsync();
        var page = await browserContext.NewPageAsync();

        page.GotoAsync("https://localhost/");
        page.ClickAsync("body > header > nav > button");

        await Task.Delay(10000); // This is just for debugging purposes
    }
);

Description

I have a Blazor Server App that is hosted in IIS. In this App, I have a Button (HTML) that has a Click handler () => NavigationManager.NavigateTo(url, true);. All the handler does is navigate to a Login Page.

I wanted to test my Application with Playwright and it works OK when I start only 1-2 instances. When I try with more than 3 instances then it hangs as soon as the click page.ClickAsync("body > header > nav > button"); is executed. if I close some instances then other instances continue working as they should.

What I have noticed is that if the second parameter in NavigationManager.NavigateTo(url, true); is set to false then it works.

This can be reproduced with the default Blazor Server Template.

P.S.

If I just run the App with dotnet run then it works!

[aslushnikov]

What I have noticed is that if the second parameter in NavigationManager.NavigateTo(url, true); is set to false then it works.

@dardanlibrary so it looks like setting it to true hits the server, and somehow the server is blocked and doesn't respond.

Do you run your blazor server in the same process with Playwright automation?

[dardanlibrary]

@dardanlibrary so it looks like setting it to true hits the server, and somehow the server is blocked and doesn't respond.

Setting that parameter to true makes the app completely reload/refresh navigation, which is required in my case. But it does not matter whether that parameter is set to true or false, it always hits the server.

Do you run your Blazor server in the same process with Playwright automation?

No, they are both separate processes and separate code projects.

[aslushnikov]

@dardanlibrary I see, I'm out of ideas then. cc'ing @mxschmitt who might have some ideas.

[mxschmitt]

I see that you never await these two calls, it should be:

        await page.GotoAsync("https://localhost/");
        await page.ClickAsync("body > header > nav > button");

Otherwise I don't see something which is wrong, if you could submit a minimal repro, we'd be happy to take a look.

[dardanlibrary]

@mxschmitt thanks for the swift reply.

The missing await part was my mistake when I was trying to remove the clutter from the code. The actual code has it.

Here you can find the minimal repo.

There is a Blazor Server project there BlazorTests that has to be published and put on IIS. There is also a Test project there BlazorTests.E2ETests where you can find two tests. One of them passes because it starts with only two Tasks and the other with four Tasks fails.

Let me now if you need something else from me.

[mxschmitt]

Oh you are using XUnit, this is discouraged, because their parallelism mode is not compatible with Playwright, see here: https://playwright.dev/dotnet/docs/test-runners#xunit-support

Could you use Nunit or MSTest instead? There it works most likely.

[dardanlibrary]

Just tried it with NUnit and I got the same result. I have also updated the repo for you to check. I have previously checked the link you provided, but the way I understand it, it does not cover my case. Based on the link, the Test Runner will run tests in parallel but they have to be separate tests. What I want, is to run the same test many times.

[mxschmitt]

I tried to reproduce it, but for me it was working fine. I'd adjust the parallelism model a bit tho, so you don't do CreateAsync all the time. The best would be that you have N workers, e.g. 3 and inside of them you create Playwright instances. These workers stay alive and then work on your jobs over and over again.

This is very similar to what we do in our nunit harness: https://playwright.dev/dotnet/docs/test-runners

[dardanlibrary]

I tried to reproduce it, but for me it was working fine. Did you host it in IIS?

I'm not sure that the proposed solution will work for my case but I will try it according to the provided link. The reason why I think that it will not work for my case is that the scenario I want to cover is to have let's say 20 users doing the same thing at the same time. To do that, I create one Playwright instance with CreateAsync and then with Parallel.ForEachAsync I create 20 Browser Contexts by using NewContextAsync and 20 Pages by usingNewPageAsync.

Since this is not working with IIS for me but it is working when I run the application with dotnet run I guess this has to suffice for now. I just cannot figure out whether this is an issue with Blazor Server or with Playwright.

[mxschmitt]

Sorry for the late reply. I'm unfortunately not familiar with Blazor either but, happy to accept any PRs/research in that direction.

[dardanlibrary]

I've been experimenting with this since my last comment. It seems the issue is on the IIS side. I was using IIS on Windows Home and not on Windows Server and based on some online documentation, IIS on Windows Home has a limit on the number of concurrent connections.

I have created a virtual machine with Windows Server and I am not facing the same issue - I was able to execute 100 parallel tests.

Sorry if I bothered you and thanks a lot for your time.