search

WordPress / wordpress-playground

Run WordPress in the browser via WebAssembly PHP
https://w.org/playground/
GNU General Public License v2.0
1.61k stars 240 forks source link

Consider Playwright for E2E testing #885

Open eliot-akira opened 8 months ago

eliot-akira commented 8 months ago

Regarding:

Recently I read some articles on this topic, and learned that both WordPress core and Gutenberg team are moving to Playwright for their E2E tests. I don't have much experience or opinion on the matter, just learning to set things up for my own projects, but from what I've read, it seems Playwright has some advantages over Cypress.

Here's the announcement post about the migration plan from existing test setup using Puppeteer.

Here's the ongoing project to migrate tests in Gutenberg.

A couple of articles comparing the two.

Cypress vs Playwright: A Detailed Comparison

Migrating from Cypress to Playwright

Some of the claimed benefits of Playwright: better performance, developer experience; intercepting native input events, file up/download, network activity; support for multiple browser engines, mobile device emulation.

I think the fact that WordPress core and Gutenberg team are using Playwright is in itself an argument for choosing it over Cypress, so that contributors to those projects don't have to learn another testing framework to be able to contribute to the Playground project.

Since the E2E tests for Playground have just been added, it seems like a good opportunity to consider an alternative, before there's a whole suite of tests.

adamziel commented 8 months ago

Thank you for bringing this up @eliot-akira! I was very torn about Playwright vs Cypress so I tried building these E2E tests using both libraries to see what's the difference in practice. Here's my Playwright branch.

Consider this test written in Playwright and then in Cypress:

Playwright

test('Switches WordPress version to ' + version, async ({ page }) => {
    await page.goto('/');
    await setWordPressUrl(page, '/wp-admin');
    // Update settings in Playground configurator
    const configurator = await page.waitForSelector('button#configurator');
    configurator.click();

    const wpVersionSelect = await page.waitForSelector('select#wp-version');
    await wpVersionSelect.selectOption(version);
    await page.click('#modal-content button[type=submit]');
    // Wait for the page to finish loading
    await page.waitForURL(new RegExp(`&wp=${version}`));

    // Go to wp-admin
    const wpFrame = await wordPressIFrame(page);
    if (version === 'nightly') {
        const footer = await wpFrame.waitForSelector('#footer-upgrade');
        expect(await footer.textContent()).toContain(
            'You are using a development version'
        );
    } else {
        await wpFrame.waitForSelector(
            'body.branch-' + version.replace('.', '-')
        );
    }
});

Here's what I like about Playwright:

Here's what I didn't like about the Playwright version of those tests:

Let's also consider some of the Playwright upsides from that comparison article:

Cross-Browser Support

Cypress also seems to support Firefox, Chrome, Edge, and other browsers.

Network Activity Interception

Cypress seems to support that as well.

Mobile Device Emulation

Cypress seems to be limited to the viewport size emulation here. It can perhaps be extended, but it sounds like Playwright is better equipped for that.

File Download and Upload

Playground has a Cypress test for exporting and importing a Playground instance via zip download/upload. It crashes in GitHub CI because of memory constraints, though. I think it can be fixed, but it would be nice not to have to deal with that.

Cypress

let versionMessage = 'Version ' + version;
if (version === 'nightly') {
    versionMessage = 'You are using a development version';
}
it('should switch WordPress version to ' + version, () => {
    // Update settings in Playground configurator
    cy.get('button#configurator').click();
    cy.get(`select#wp-version option[value="${version}"]`).should(
        'exist'
    );
    cy.get('select#wp-version').select(`${version}`);
    cy.get('#modal-content button[type=submit]').click();
    // Wait for the page to finish loading
    cy.url().should('contain', `&wp=${version}`);

    // Go to phpinfo
    cy.setWordPressUrl('/wp-admin');
    cy.wordPressDocument()
        .find('#footer-upgrade')
        .should('contain', versionMessage);
});

Here's what I like about Cypress:

Here's what I didn't like about Cypress:

Let's also consider some of the Cypress shortcomings from that comparison article:

Inability to instantiate multiple browsers simultaneously Lack of support for multi-tab testing.

I don't see a good use-case for doing either of those in the Playground test suite. Perhaps one will emerge in the future, but I don't anticipate that today.

JavaScript is the preferred language for writing test cases using Cypress.

Cypress seems to have great support for types, the existing test suite is all TypeScript.

Conclusion

Cypress was a clear winner to me. At the same time, we don't have many E2E tests yet, as you noticed. There may still be very good reasons to switch to Playwright and I'm very open to that possibility. At the moment, though, I'm more convinced about Cypress.

eliot-akira commented 8 months ago

Wow, that's a deep dive comparing Playwright and Cypress, considering them from various aspects, pros and cons. That's true, the latter test does read nicely, compact with natural flow of actions.

What stood out from the arguments against Playwright: in addition to taking longer to write the tests (fluency and intuitiveness):

couldn't get it to reliably work with the nested iframes structure.. tried for a few hours..

That sounds like the decisive point, if it doesn't work well for the specific needs of the Playground project. I think that concludes the matter, since there's no use fighting the tool when there's already a working testing setup started.

adamziel commented 8 months ago

Sounds good! I'll close this issue for now, then.

adamziel commented 8 months ago

Actually, @wunderbart shared this version of the Playwright test that looks much more appealing:

test( 'Switches WordPress version to ' + version, async ( { page } ) => {
    await page.goto( '/' );
    await setWordPressUrl( page, '/wp-admin' );

    // Update settings in Playground configurator
    await page.locator( 'button#configurator' ).click();
    await page.locator( 'select#wp-version' ).selectOption( version );
    await page.locator( '#modal-content button[type=submit]' ).click();

    // Go to wp-admin
    const wpFrame = page.frameLocator( 'iframe' );
    if ( version === 'nightly' ) {
        await expect( wpFrame.locator( '#footer-upgrade' ) ).toContainText(
            'You are using a development version'
        );
    } else {
        await expect(
            wpFrame.locator( 'body.branch-' + version.replace( '.', '-' ) )
        ).toBeVisible();
    }
} );

So maybe let's give Playwright another chance provided it can reliably handle the doubly nested iframes used in Playground!

WunderBart commented 8 months ago

So maybe let's give Playwright another chance provided it can reliably handle the doubly nested iframes used in Playground!

It should be pretty straightforward by using frameLocator! You can pierce any number of iframes with it, like this:


const block = page
    .frameLocator( 'iframe#parent-frame' )
    .frameLocator( 'iframe#child-frame' )
    .frameLocator( 'iframe#grandchild-frame' )
    // etc.
    .locator( '.block' );

await block.click();
adamziel commented 7 months ago

Okay, I'm on board. I would love to give Playwright one more try! I don't have any bandwidth to actually explore that – any volunteers? :-)

adamziel commented 14 hours ago

Additional reasons to use Playwright: