[Feature] Allow test run to continue the case execution if a test.step() fails

[gegoncalves]

It would be nice to have a flag or something that could be use to allow a test run go trough all the test.step(), even if any of them fail.

Something like: test.describe.configure({ mode: 'continue' }); 😄

The reason for this request is explained in this comment from the first line until the first bullet point: https://github.com/microsoft/playwright/issues/14508#issuecomment-1144617999

[aslushnikov]

@gegoncalves You can use expect.soft inside the steps, and they will continue running. Would it work for you?

[gegoncalves]

@aslushnikov unfortunately no. If was only the assertion that would be nice. However, if there is any timeout before the other steps, it is going to stop the run.

nonsense code example: 4th line gonna timeout and stop the execution

import { test, expect } from '@playwright/test';

test('basic test', async ({ page }) => {
    await test.step('Test 1 Not fail', async () => {
        await page.goto('https://playwright.dev/');
        await page.locator('.navbar__inner .sdfdsfsdf').click();
        const title = page.locator('.navbar__inner .navbar__title');
        await expect(title).toHaveText('Playwright');
    });

    await test.step('Test 2 pass', async () => {
        await page.goto('https://playwright.dev/');
        const title = page.locator('.navbar__inner .navbar__title');
        await expect(title).toHaveText('Playwright');
    });

    await test.step('Test 3 should fail', async () => {
        await page.goto('https://playwright.dev/');
        const title = page.locator('.navbar__inner .navbar__title');
        await expect(title).toHaveText('blabla');
    });
});

[aslushnikov]

nonsense code example: 4th line gonna timeout and stop the execution

@gegoncalves ah, yes. In this case you'll have to fallback to using tests instead of steps. There are no plans to make test execution continue on step failure (aka test.step.soft)

[gegoncalves]

Ah ok. That way I will be back to the problem of multiple instances of browsers being used from the issue https://github.com/microsoft/playwright/issues/14508#issuecomment-1144617999

I need to figure out how to reuse single page between tests using custom Fixtures. Unfortunately, is not possible to do using context or page.

Here is a stupid code question: @aslushnikov Is it possible to do this code snippet on my basePage class?

Document snippet

let page: Page;

test.beforeAll(async ({ browser }) => {
  page = await browser.newPage();
});

My base actions exported into other PO:

export class baseActions{
    readonly page: Page;

    constructor(page: Page) {
        this.page = page;
    }

    async navigateToURL(url: string) {
        await this.page.goto(url);
    }

    async waitForElementAttached(locator: string): Promise<void> {
        await this.page.locator(locator).waitFor();
    }
...

[aslushnikov]

Here is a stupid code question: @aslushnikov Is it possible to do this code snippet on my basePage class?

@gegoncalves Yes, definitely - you should just have the baseActions instance next to your page variable. Like this:

let page: Page;
let actions: baseActions;

test.beforeAll(async ({ browser }) => {
  page = await browser.newPage();
  actions = new baseActions(page);
});

[gegoncalves]

Is it possible to use instead of a global-setup a "local-setup" based on this: https://playwright.dev/docs/test-auth#multiple-signed-in-roles

The way the doc explain the usage, it is going to be used across all the spec.ts files. That can cause a lot of false negatives/positives in case the cookies has wrong info

[aslushnikov]

The way the doc explain the usage, it is going to be used across all the spec.ts files.

@gegoncalves Why so? You can specify the storageState option only for the tests / suites / spec files where you need it:

import { test } from '@playwright/test';

test('regular test - no cookies', async ({ page }) => {
});

test.describe('signed in tests', () => {
  test.use({ storageState: 'userStorageState.json' });

  test('user test', async ({ page }) => {
    // page is signed in as a user.
  });
});

[gegoncalves]

Thanks @aslushnikov and this is my last question to close that issue:

// global-setup.ts
import { chromium, FullConfig } from '@playwright/test';

async function globalSetup(config: FullConfig) {
  const browser = await chromium.launch();
  const adminPage = await browser.newPage();
  // ... log in
  await adminPage.context().storageState({ path: 'adminStorageState.json' });

  const userPage = await browser.newPage();
  // ... log in
  await userPage.context().storageState({ path: 'userStorageState.json' });
  await browser.close();
}

export default globalSetup;

Is there a way to not specify a browser? On this line const browser = await chromium.launch(); we always need to choose one.

I want to use the --project command instead of have to do this for multiple browsers

[pavelfeldman]

It is not currently possible because global setup is global and spans across the projects. You can pass an environment variable and use it in global setup to pick a browser of liking as you are running your tests though.

[gegoncalves]

It is not currently possible because global setup is global and spans across the projects. You can pass an environment variable and use it in global setup to pick a browser of liking as you are running your tests though.

I got it. Thanks @pavelfeldman

Just to confirm: @aslushnikov there is no plan for a test.step.soft()? If so, I am gonna close that ticket.

And again thanks for the help and amazing work that you have been doing with playwright

[aslushnikov]

Just to confirm: @aslushnikov there is no plan for a test.step.soft()?

@gegoncalves Yep, no plans for this so far.

And again thanks for the help and amazing work that you have been doing with playwright

Thank you for the warm words! ❤️

[sagar-goldcast]

Anyone working on original feature ask? I think thread is about : "Allow test run to continue the case execution if a test.step() fails."

I am confirming again that, this is the ask.

import { test, Page } from '@playwright/test';

test.describe.configure({ mode: 'serial' });

let page: Page;
let actions: generalActions;

test.beforeAll(async ({ browser }) => {
  page = await browser.newPage();
  actions = new generalActions(page);
});

test.afterAll(async () => {
  await page.close();
});

test('runs first', async () => {
  await actions.navigateToURL('https://playwright.dev/'); // failed step here.
});

test('runs second', async () => {
  await actions.clickElement('text=Get Started');
});

Playwright is not giving any options to continue test after "first" test fails. I have long marathon flows for e2e test.

[pradeipp]

Commenting for reach. Would love to have this feature during long e2e tests with very peculiar conditions

[harshkhare3]

Agreed! We need this feature please for e2e tests

[raphatmr]

Same problem here. So we are going to use our own wrapper function for steps:

import { test, expect } from '@playwright/test';

export type StepOptions = {
    soft: boolean;
};
export type StepBody<T> = () => T | Promise<T>;
export function step<T>(title: string, body: StepBody<T>): Promise<T>;
export function step<T>(title: string, options: StepOptions & { soft: false }, body: StepBody<T>): Promise<T>;
export function step<T>(title: string, options: StepOptions, body: StepBody<T>): Promise<T | undefined>;
export async function step<T>(title: string, optionsOrBody: StepOptions | StepBody<T>, body?: StepBody<T>): Promise<T | undefined> {
    const isSoft = typeof optionsOrBody === 'function'
        ? false
        : !!optionsOrBody.soft;
    const actualBody: StepBody<T> = typeof optionsOrBody === 'function'
        ? optionsOrBody
        : body!;
    return await test.step(title, async () => {
        if (isSoft) {
            try {
                return await actualBody();
            } catch (error) {
                /* Mark the test as failed, but continue. */
                /* '<unknown error>' in case someone thinks 'throw undefined' is a good idea. */
                /* Try to build a message, containing the original stack trace.
                 * From what I have observed so far, 'error.stack' also contains a message, but we also provide some fallback here,
                 * as we cannot expect 'error' to always have certain properties. */
                expect.soft(error ?? '<unknown error>', `Step "${title}" failed: ${error.stack || error.message || error || '?'}`)
                    .toBeUndefined();
                return undefined;
            }
        }
        return await actualBody();
    });
}

The step can be marked as soft via the optional options. If any error is thrown in the step body, the test will be marked as failed is not terminated (this is done with some dummy soft assertion, that will always fail). Soft steps may return undefined even if the step body will never return undefined values. By default, steps are non-soft. However, depending on the requirements, it might be better to make them soft by default.

/* non-soft by default */
const result1: string             = await step('step 1',
                                               async () => { /* ... */ return 'result 1'; });
/* explicitly marked as non-soft */
const result2: string             = await step('step 2',
                                               { soft: false },
                                               async () => { /* ... */ return 'result 2'; });
/* marked as soft (may return undefined) */
const result3: string | undefined = await step('step 3',
                                               { soft: true },
                                               async () => { /* ... */ return 'result 3'; });

Is this a valid thing to do, or do we have to expect any problems, when using this in our tests?

[mikaelkarlsson-se]

+1 about adding soft for steps (or something similar)! 🥎 In our case, we have a SAAS solution on which we run snapshot tests for each of our clients. When we change something in the core of our product this might affect several clients and/or several places (pages) in the application. But since the tests stop after an error, we are unable to gather a complete summary which would have helped us a lot when taking necessary actions. As of now, we must run the tests over and over and since we aren't getting the complete overall picture until late in the process we risk making bad decisions along the way.