[REGRESSION]: Run time doubled after upgrade to 1.32.x

[spaniardmaximus]

Context:

• GOOD Playwright Version: 1.31.2
• BAD Playwright Version: 1.32.x
• Operating System: Windows
• Extra: Environment: Windows Server 2019, web server: nginx, tests run headless. We tried 1.32.x with Chrome v111 and no difference to eliminate v112 from the issue.

Describe the bug We have some 800+ tests running via a Jenkins job against our app in a Windows environment using Chromium. Test run was around 40 min. Context remaining the same and after upgrading to 1.32.2, we noticed the runtime go up to 2.5h approximately. With the latest version (1.32.3), the runtime has improved but still far from ideal - 1.5h.

The test run remains successful but the increase in time is a major drag in our workflow. We cannot see any other change than the Playwright version. We tracked the build times of our app and it was the same. Only the task of running the tests spiked massively in time.

[spaniardmaximus]

I appreciate the vagueness of it all. No code whatsoever but my colleagues and I can only track the problem to the Playwright version. In fact, I should have mentioned in the description, we have "fixed" the job to run 1.31.2 for the foreseeable not to fall in the massive runtime increase trap.

[mxschmitt]

Are you using Google Chrome channel or using our Chromium?

Are you using sharding?

We need unfortunately a repro, otherwise its hard for us to act on it.

You can try to set DEBUG=pw:api or record a Playwright trace to spot actions which take much longer than before.

[spaniardmaximus]

Thanks @mxschmitt for getting back to us.

• We use this project in playwright.config.ts:

{ name: 'Desktop-Chromium', use: { ...devices['Desktop Chrome'] } }

• We do not use sharding
• I'll try to get to a test and setup that I can share to reproduce the issue. Meanwhile, I'll look into the debugging and trying to find the source of the slowing down.

[mxschmitt]

We tried 1.32.x with Chrome v111 and no difference to eliminate v112 from the issue.

How did you do this? By setting use: { launchOptions: { executablePath: ... }}?

[neilsleightholm-paxton]

We did this using the "channel" option:

    {
        name: 'Desktop-Chrome',
        use: { channel: 'chrome' }
    },
    {
        name: 'Desktop-ChromeBeta',
        use: { channel: 'chrome-beta' }
    },

[mxschmitt]

Sounds good as well which will use the same Google Chrome between your good and bad Playwright version.

So we need unfortunately a reproduction to be able to dig into it, it can be also shared privately with us over e.g. Discord.

[spaniardmaximus]

My colleague @neilsleightholm-paxton has tried several combinations of browser/playwright versions and 1.32.x seems to be the cause of the catastrophic slow down of our tests.

@mxschmitt , we've putting something together to show you. Is there a way to show you securely and in confidence?

[mxschmitt]

You can ping me on Discord if you are not comfortable sharing your project on GitHub. You can also write me an email: maxschmitt @ microsoft.com

[aslushnikov]

(over to Max since he's a touch point in this discussion)

[spaniardmaximus]

UPDATE: My colleague ran with v1.33 and runtime returned to original time/speed. I don't know if there is something obvious on this diff https://github.com/microsoft/playwright/compare/release-1.32...release-1.33. I wouldn't know what area to focus on that might have affected speed for us.

[mxschmitt]

Glad to hear! If you are interested, you could try bisecting with our nightly releases. See here for all our nightly releases: https://www.npmjs.com/package/@playwright/test?activeTab=versions

Did you by accident run them in debug mode / with Inspector attached? There we fixed a regression which got introduced in a similar date range.

[spaniardmaximus]

Hi Max, I doubt that it was that. The command to run the tests is fixed and we don't offer a flag in Jenkins to change it to debug mode.

[spaniardmaximus]

We're working on this...

Glad to hear! If you are interested, you could try bisecting with our nightly releases. See here for all our nightly releases: https://www.npmjs.com/package/@playwright/test?activeTab=versions

I'd be very interested to see if there is that one build that went from "not working" to "working" for us. I'll let you know.

[neilsleightholm-paxton]

Working through the versions we have found the following: 1.33.0-alpha-apr-11-2023 - Good 1.33.0-alpha-apr-10-2023 - Slow

[mxschmitt]

Great range!

Diff: https://github.com/microsoft/playwright/compare/2b32d051752976720f6c8d952974edd625e503bb...1e9668f8bd2bdd402df57163c097797c0a7ccc83

I believe its 2af3f486c4f141400d57912ae54785c5455757c0 which sounds suspicious. I believe you had tracing on?

[neilsleightholm-paxton]

We didn't intentionally have tracing on, how would that be enabled? Or are you suggesting it was enabled by default?

[spaniardmaximus]

Thanks for the investigative work @neilsleightholm-paxton. @mxschmitt, this is our config setting:

trace: 'retain-on-failure',

It has always been the same. Our tests weren't failing, they were taking 5 times longer to run.

[mxschmitt]

"retain-on-failure" means that for each test, a trace is recorded, but only saved when its failing. We recommend "on-first-retry" instead. This will only record it when its failing, on the first retry. Make sure to have retries: process.env.CI ? 2 : undefined configured as well, so you have retries enabled on your CI.

How big are your traces? You can change trace to "on" to see how large they are (from the test-results folder), this might confirm our theory about this performance regression. For larger websites or testing scenarios, trace files are larger, so it gets then slower if you have traces enabled all the time (aka. retain-on-failure).

[spaniardmaximus]

Thanks for the advice and your time on this @mxschmitt. I'll make sure that I implement the changes.

[spaniardmaximus]

@mxschmitt , your advised changes against the problematic version (1.32.3) worked perfectly, i.e. runtime was as expected. We'll keep the changes and make the jump to 1.33 anyway. Thanks.