[Feature]: Add the ability to configure all assertions as soft in playwright config file

[gal-checksum]

🚀 Feature Request

Add the ability to configure all assertions as soft in playwright config file.

When you run tests in your CI/CD, you want assertions to be "hard" as there is no point to continue running a test if the assertion failed.

However, when debugging/authoring a test, it'll be useful to use soft assertions, so you can run the full test and fix all of it. This also recommended in the Playwright best practices: https://playwright.dev/docs/best-practices#use-soft-assertions

So I think there should be an easy way to make all assertions soft temporarily or based on the environment. Similar to how you can configure timeouts to all of the assertions.

Example

No response

Motivation

Ability to debug tests with soft assertions so debugging is faster but still run tests with hard assertions

[symon-skelly1991]

Hi, I created a utility file that can do this while this is being investigated (it's a very useful request):

//my-test.ts
import { test, expect as baseExpect, type Expect } from '@playwright/test'
test.beforeEach(async () => {
  if (process.env.SOFT === 'True') { enableSoftAssertions() }
})

test.afterEach(async ({ }, testInfo) => {
  if (process.env.SOFT === 'True') { assertAllSoftAssertions() }
}

// Global storage for soft assertion errors
let softAssertionErrors: any[] = []
let useSoftAssertions = false // Flag to enable/disable soft assertions

// Function to enable soft assertions
function enableSoftAssertions () {
  useSoftAssertions = true
}

// Function to disable soft assertions
function disableSoftAssertions () {
  useSoftAssertions = false
}

// Function to assert all soft assertions
function assertAllSoftAssertions () {
  if (softAssertionErrors.length > 0) {
    const errors = softAssertionErrors.map(err => err.toString()).join('\n\n')
    softAssertionErrors = [] // Reset errors
    throw new Error(`Soft assertion errors:\n${errors}`)
  }
}

// Creating a proxy for the expect function to intercept all methods
function createExpectProxy (baseExpect: Expect) {
  return new Proxy(baseExpect, {
    apply (target, thisArg, argumentsList) {
      const originalResult = Reflect.apply(target, thisArg, argumentsList)
      if (!useSoftAssertions) {
        return originalResult
      }
      return new Proxy(originalResult, {
        get (target, prop, receiver) {
          const originalMethod = Reflect.get(target, prop, receiver)
          if (typeof originalMethod === 'function') {
            return (...args: any[]) => {
              try {
                return originalMethod.apply(target, args)
              } catch (error) {
                softAssertionErrors.push(error)
                // Return a dummy passing result to keep the test going
                return {
                  pass: true,
                  message: () => `Soft assertion failed: ${error.message}`
                }
              }
            }
          }
          return originalMethod
        }
      })
    }
  })
}

let expect = baseExpect
// Wrap the original expect with our proxy if SOFT is True
if (process.env.SOFT === 'True') {
  expect = createExpectProxy(baseExpect)
}

export { expect }

You just set SOFT as an env var when running your tests and it will do as you've asked.

Or, if you prefer, you can set it inside of your playwright config:

process.env.SOFT = process.env.SOFT ?? 'False' or process.env.SOFT = process.env.SOFT ?? 'True'

Of course, you need to import expect from this file inside of your spec files for it to work.

[cctui-dev]

Big plus 1 for this