[BUG] When on serial mode, assertions prevent subsequent tests to run.

[gal064]

## System:
 - OS: macOS 11.4
 - Memory: 52.63 MB / 8.00 GB
## Binaries:
 - Node: 17.8.0 - /usr/local/bin/node
 - Yarn: 1.22.18 - /usr/local/bin/yarn
 - npm: 8.5.5 - /usr/local/bin/npm
## Languages:
 - Bash: 3.2.57 - /bin/bash
## npmPackages:
 - playwright: ^1.24.2 => 1.24.2 

Describe the bug:

1. Run multiple tests on serial mode
2. In one of the tests, have a soft assertion that fails
3. Expected (e.g. correct) result: The specific test is marked as failed but continues to run. And, subsequent tests continue to run as well and if successful marked as passed
4. Actual (e.g. incorrect) result: The specific test is marked as failed but continues to run But subsequent tests do not run

Code Snippet:

Help us help you! Put down a short code snippet that illustrates your bug and that we can run and debug locally.

const { test, expect } = require('@playwright/test');

test.beforeEach(async ({ page }) => {
  await page.goto('https://demo.playwright.dev/todomvc');
});

const TODO_ITEMS = [
  'buy some cheese',
  'feed the cat',
  'book a doctors appointment'
];

test.describe.serial('New Todo', () => {
  test('should allow me to add todo items', async ({ page }) => {
    // Create 1st todo.
    await page.locator('.new-todo').fill(TODO_ITEMS[0]);
    await page.locator('.new-todo').press('Enter');

    ########## This soft assertion will prevent next tests to run because the test runs on serial mode ############
    expect.soft(2).toEqual(1)

    // Make sure the list only has one todo item.
    await expect(page.locator('.view label')).toHaveText([
      TODO_ITEMS[0]
    ]);

    // Create 2nd todo.
    await page.locator('.new-todo').fill(TODO_ITEMS[1]);
    await page.locator('.new-todo').press('Enter');

    // Make sure the list now has two todo items.
    await expect(page.locator('.view label')).toHaveText([
      TODO_ITEMS[0],
      TODO_ITEMS[1]
    ]);

    await checkNumberOfTodosInLocalStorage(page, 2);
  });

  test('should clear text input field when an item is added', async ({ page }) => {
    // Create one todo item.
    await page.locator('.new-todo').fill(TODO_ITEMS[0]);
    await page.locator('.new-todo').press('Enter');

    // Check that input is empty.
    await expect(page.locator('.new-todo')).toBeEmpty();
    await checkNumberOfTodosInLocalStorage(page, 1);
  });

  test('should append new items to the bottom of the list', async ({ page }) => {
    // Create 3 items.
    await createDefaultTodos(page);

    // Check test using different methods.
    await expect(page.locator('.todo-count')).toHaveText('3 items left');
    await expect(page.locator('.todo-count')).toContainText('3');
    await expect(page.locator('.todo-count')).toHaveText(/3/);

    // Check all items in one call.
    await expect(page.locator('.view label')).toHaveText(TODO_ITEMS);
    await checkNumberOfTodosInLocalStorage(page, 3);
  });

  test('should show #main and #footer when items added', async ({ page }) => {
    await page.locator('.new-todo').fill(TODO_ITEMS[0]);
    await page.locator('.new-todo').press('Enter');

    await expect(page.locator('.main')).toBeVisible();
    await expect(page.locator('.footer')).toBeVisible();
    await checkNumberOfTodosInLocalStorage(page, 1);
  });
});

/**
 * @param {import('@playwright/test').Page} page
 * @param {number} expected
 */
 async function checkNumberOfTodosInLocalStorage(page, expected) {
  return await page.waitForFunction(e => {
    return JSON.parse(localStorage['react-todos']).length === e;
  }, expected);
}

/**
 * @param {import('@playwright/test').Page} page
 * @param {number} expected
 */
 async function checkNumberOfCompletedTodosInLocalStorage(page, expected) {
  return await page.waitForFunction(e => {
    return JSON.parse(localStorage['react-todos']).filter(i => i.completed).length === e;
  }, expected);
}

/**
 * @param {import('@playwright/test').Page} page
 * @param {string} title
 */
async function checkTodosInLocalStorage(page, title) {
  return await page.waitForFunction(t => {
    return JSON.parse(localStorage['react-todos']).map(i => i.title).includes(t);
  }, title);
}

[mxschmitt]

Sounds like a feature request to me. The current definition of soft assertions are that they will get collected and thrown on the test level: https://playwright.dev/docs/test-assertions#soft-assertions

To have them collected and thrown at the last test when using serial mode is very different.

[gal064]

Yep soft assertions may work according the dry definition but I think it's a bug from a user perspective (i.e. a miss when defining soft assertions).

The intended use-case of soft assertions is for a failure to not stop the test suite from running. This is not the case with serial tests, and probably not intentionally. I don't see any logic to why the current test should continue to run but subsequent tests stop.

With that said, I understand if it's not top priority though

[RaviThippana]

Any update on the above feature/bug. Soft Assertions in serial mode, when we have multiple tests

[jamalebp-formant]

Any update on the above feature/bug. Soft Assertions in serial mode, when we have multiple tests

Also looking for an update on this

[michelle-choi1]

Yep soft assertions may work according the dry definition but I think it's a bug from a user perspective (i.e. a miss when defining soft assertions).

The intended use-case of soft assertions is for a failure to not stop the test suite from running. This is not the case with serial tests, and probably not intentionally. I don't see any logic to why the current test should continue to run but subsequent tests stop.

With that said, I understand if it's not top priority though

agreeing with @gal064 - my use case is that I'm using serial mode to split filling out an order checkout process (which includes a lot of steps, expects, and page navigations) into multiple tests that have to run sequentially, and i just ran across a case in which i'd like an assertion to be soft (not to keep the soft failure permanently, but as a temporary measure to help me speed development) but i also understand there are other priorities. just wanted to add a data point!

[sagar-goldcast]

I hope team is working on this bug. Any PR or work has been done so far ? Still see this in new version.

[Nantris]

This would be really helpful. I'd love to know all the failing tests if there's a failing test but in serial mode (practically required for Android) a fail stops the tests.

[jaktestowac]

Allowing the serial mode test to continue even if one test fails would be a very nice feature💪

[success-stha]

This feature would be really helpful

[BagchiMB]

This is a very required feature, would love to have this thing.

[dgozman]

We recommend to wrap your expects and check that no errors were thrown in the last test.

For example, introduce these two helpers:

const errors: any[] = [];
async function soft(cb: () => any) {
  try {
    await cb();
  } catch (e) {
    errors.push(e);
  }
}
async function checkErrors() {
  await test.step('check errors', async () => {
    for (const e of errors)
      expect.soft(() => { throw e }).not.toThrow();
  }, { box: true });
}

Now, you can use these helpers to wrap your expects, and check errors in the very last test:

test.describe.configure({ mode: 'serial' });

test('test1', async ({ page }) => {
  // ...
  await soft(() => expect(page.locator('body')).not.toBeVisible());
  // ...
});

test('test2', async ({ page }) => {
  // ...
  await soft(() => expect(1).toBe(2));
  // ...
});

test('check errors', async () => {
  await checkErrors();
});