stretchr / testify

A toolkit with common assertions and mocks that plays nicely with the standard library
MIT License
23k stars 1.58k forks source link

Require should support parallel execution, and should not abort all tests in case of failure #1623

Open priyankshah217 opened 2 months ago

priyankshah217 commented 2 months ago

Description

I am testing a workflow, so if one test step fails, the test should not proceed. For example, if the login fails, there is no point in adding items to the cart. I am using assertions from the 'require' package. It works fine for a handful of tests, but as the number of tests grows, parallel execution seems to be the only option. However, when we made these tests run in parallel, they failed at very basic checks. I understand that in "require," if an assertion fails, we call the FailNow method, which stops execution. Ideally, it should only stop the test/goroutine that failed this assertion and not affect other running tests/goroutines.

Here is basic code snippet with testify https://go.dev/play/p/7a8mh9eVO6r

Without testify (Using native go testing framework) https://go.dev/play/p/9NxHqKpmPam

Proposed solution

When tests are run in parallel, it's crucial that a required assertion failing does not disrupt other ongoing runs.

Use case

I'm phasing out parallel runs and considering switching to a native test framework. We're thinking of using Testify, and I may create a PR if time allows.

jhoppenstedt commented 1 month ago

Ideally, it should only stop the test/goroutine that failed this assertion and not affect other running tests/goroutines.

I observed a similar behaviour when calling suite.T().Skip("") which also fails the whole suite.. Code snippet: https://go.dev/play/p/p_Q71r_hP2S

Antonboom commented 2 weeks ago

Suite doesn't support t.Parallel()

Covered by testifylint#suite-broken-parallel