[Feature] Test filter API so we can programmatically pick which tests to run

[dylang]

Hi!

Problem

We don't have a programmatic way to pick which tests are used.

Background

We have a test tracking system (Zephyr for Jira) that we use to group test in various ways, such as by feature.

• If all tests for a feature pass, that feature is ready to ship.
• If a change is made to a feature, all tests for that feature must be re-run.

How this is handled today with Playwright

Every test file has a ticket number in the filename.

We manually look in the test tracking system for, say, the 7-100 tickets associated with a feature we are interested in, and put those on the command line or in the CI system:

npx playwright test E2E-4841 E2E-3433 E2E-1112 E2E-1114 E2E-1116 E2E-3331 E2E-2233 

This is tedious and error-prone.

Proposed Solution

A programmatic API for filtering the available tests.

The API will provide an array of all tests found by Playwright based on the existing command line and project settings. The function returns a promise of the same shape array of tests, but not necessary all of the tests need to be included.

It is up to developers using the API to determine how they want to handle the filtering.

Simplified example:

# Run all test related to feature-1024
TEST_FEATURE_ID=FEAT-1024 yarn test

Custom Filter code:

// This function is provided all tests that Playwright Test finds using the project config.
async customFilter(tests) {
    // Get the user input. There could be other options too, like category, smokeTest, severity.
    const { TEST_FEATURE_ID } = process.env;

    if (!TEST_FEATURE_ID) {
      // Not specifying which feature? Then just run all tests. 
      return tests;
    }

    // Get all the tickets based on what the user asked for. 
    // This might use a REST api. It's not Playwright-specific.
    const { tickets } = await getTicketsForFeature(TEST_FEATURE_ID);

    // Only include tests that have a filename that includes one of the ticket numbers.
    const testsToRun = tests.filter(({ testFileName }) => tickets.some(({ ticketNumber }) => testFileName.includes(ticketNumber)));

    // Now, instead of running all tests, we're just running tests that related to TEST_FEATURE.
    return testsToRun;
  }

Prior work

We're doing this today in Jest using their Custom Sequencer API.

[dylang]

The @tag feature does not work for this because we manage the equivalent of tags in the external test manager (Zephyr for Jira).

For example, managers can use this tool to add labels, like a customer id, and we don't want them to modify the test code to do this.

[dylang]

A potentially alternative is dynamically generating the grep expression for a project because our Playwright config export can be an async function!

I'm experimenting with this now.

It's not a full fix though, because we aren't provided the list of tests, which would make this easier to implement and more powerful.

[dylang]

I'm willing to make a pull request for this feature if there is interest from the Playwright team.

[gajus]

Can we also add the ability to include fixme/skip tests?

[dylang]

Hi @gajus, I appreciate your suggestion. I'm not sure if I'll be able to implement that kind of static state change to a test. I was more thinking just simple array filtering.

[dylang]

Hi, I've figured out a workaround, so maybe it's a matter of documentation and examples:

• playwright.config.ts can return an async function. (Is this documented?)
• testMatch can be a string[].
• We use globby to find all the tests ourselves.
• We then filter the array of test fileson our test tracking system. This way we're only running the specific test files we need for that run.

Example:

E2E_TEST_CYCLE=ETOE-C4970 yarn playwright test

[gajus]

Wow, good to know!

[amtabug]

It would be great to be able to pass a predicate function to a TestProject config. Something like testTitlePredicate?: (title: string) => boolean;

I tried doing this a while ago but never got around to finishing it: https://github.com/microsoft/playwright/compare/main...amtabug:playwright:testFilterPredicate