[BUG] `test.fail(true)` will not pass if failure is caused by `.waitFor*()` function timeout

[samijaber]

Source code

• [x] I provided exact source code that allows reproducing the issue locally.

Test file (self-contained)

test('should pass', async ({ page }) => {
    test.fail(true);

    await page.goto('https://www.google.com');

    // this will cause a failure
    await page.waitForSelector('text=somerandomstring');
    // this will cause a failure
    await page.getByText('somerandomstring').locator('visible=true').waitFor();
    // this will pass
    await expect(page.getByText('somerandomstring')).toBeVisible();
  });

Steps

• Run the test
• test will fail until both .waitFor* commands are commented out

Expected

I would expect timeout errors caused by .waitFor* functions to be ignored when test.fail(true).

Actual

test.fail(true) only "allows" expect() failures to pass.

Let me know if my understanding of this feature is wrong!

[rob4629]

Do you believe setting test.fail(true) should ignore ANY failure? I feel like tests could get into a broken (/useless) state if that were the case, whereas limiting it to just assertions can be more targeted/intentional.

Just commenting to provide an opinion, and follow along with any discussion.

[mxschmitt]

This sounds unexpected, I can repro. We'll look into it.

[mxschmitt]

Turns out this is working as expected as per https://github.com/microsoft/playwright/issues/9373#issuecomment-938161218. You should make your test fail before the test timeout has reached.

waitForSelector has a timeout of 0 by default, so you should reduce it if you expect a failure. If expect() fails before the test-timeout has reached, it gets marked as passing.

[samijaber]

@mxschmitt I would like to bring this issue back up, because I still regularly struggle with it in my day-to-day.

Full context:

• I have 8 SDKs for web frameworks (Vue, React, Svelte, Solid, Qwik, etc.) that are identical in functionality
• each SDK is tested in multiple servers (Nextjs, Remix, Nuxt, Vite, etc).
• I have a large Playwright parametrized test suite running against each of the above servers that import these SDKs.

Not all servers pass all tests: I am in the process of fixing bugs/adding feature coverage. I want to track which tests are falling for which parameters, to make sure I enable the tests when the bugs are fixed. This is why I rely on test.fail(condition) instead of test.skip(condition).

However, according to this github issue, someone in my situation is unable to use the entirety of waitFor* helpers in Playwright, because then I cannot expect test failures.

This makes using playwright tricky, because I have to either:

• constantly avoid a good chunk of Playwright's API 😓
• use test.skip(isSDKFoo) and hope that I remember to enable the tests when I fix the bug for isSDKFoo

What would your advice be for someone in my situation?