search

avajs / ava

Node.js test runner that lets you develop with confidence 🚀
MIT License
20.74k stars 1.41k forks source link

AVA does not Bail after Assertion Failure with FailFast Option Enabled #2385

Closed PeterMerkert closed 4 years ago

PeterMerkert commented 4 years ago

A minimal code sample is:

const test = require('ava')

test('AVA does not stop when assertion fails', async t => {
    t.is(42, 43)
    console.log(42)
})

This will print 42 on my console, despite the first test actually failing. I do expect here, that the log output line is not executed any more.

My package.json includes:

"ava": {
    "failFast": true
  },

I assume the FailFast option is more about concurrent test runners and stopping them gracefully just after a test has failed. I seem to be unable to locate an option to actually force AVA to bail out of a single test once an assertion fails. I do not want to stop any other test executions and also not any other tests in the same file.

Is there such an option to stop the execution immediately for the single test once an assertion within the test failed?

novemberborn commented 4 years ago

Is there such an option to stop the execution immediately for the single test once an assertion within the test failed?

No. Assertions don't throw errors, so they don't cause the test to stop executing.

There are some ways of doing this but whether they make sense depend on your use case. Could you elaborate?

(I'm closing this issue for housekeeping purposes, but let's keep the conversation going.)

PeterMerkert commented 4 years ago

Thanks for helping out.

The short explanation: I would like to avoid an if-cascade to always double check. It feels like I do not need AVA assertions than any more (except of neat highlighting). So, I want to avoid:

let res = apiCall1
t.is(res.status, 200)
if (res.status !== 200) return
res = apiCall2
t.is(res.status, 200)
if (res.status !== 200) return
...

And anyways, AVA only highlights for me the first assertion failing in a test and not the subsequent ones. So, if there are two assertions failing within a single test, only the first one is shown in my output log.

The long explanation: I want to archive this is because I am running some integration tests, creating users not only in my API, but also in Auth0 as a SaaS provider for user handling. Within that test, I cross-check if emails are send out properly and if the user is created in all the correct fashions that the login works (username + password, magic email code, sms code).

In the end of the test, I want to remove the user from Auth0, as otherwise it gets cluttered with test users (though I already use a test tenant). And once a user is created with a phone number, it can not be created again. Thus, I clean up.

However, if something fails in the user creation step in the beginning (like the API returns 409), and then I do a check t.is(response.statusCode, 200) this assertion fails. However, the test execution just goes on. This will lead to longer test runtime than necessary and obviously probably a lot of subsequent failing other calls, which will also clutter any log collected during testing.

novemberborn commented 4 years ago

AVA only highlights for me the first assertion failing in a test and not the subsequent ones. So, if there are two assertions failing within a single test, only the first one is shown in my output log.

We'd like to show all failures though. See #261.

The long explanation

Right, so you're using AVA to orchestrate a more complex integration test. Which is great! But not quite what it's designed for.

Mind you, I use AVA to run integration tests, with carefully ordered serial tests, and that works great. But I'm mocking the APIs I call out to.

We have an experimental t.try() assertion you could use to maybe help with this orchestration. Unfortunately we didn't quite get the documentation done yet. There's some hints here: https://github.com/avajs/ava/releases/tag/v2.4.0.

cakoose commented 3 years ago

I have a data structure that efficiently maintains a set of ranges. To test it, I instantiate a simpler/dumber implementation along with the real implementation, perform a bunch of random operations to both, and then check that they end up the same.

When a test fails, I usually add additional logging to figure out what led to the failure.

With Ava's current behavior, it's way too hard to figure out what's going on because the logging doesn't stop at the first failure. This ended up being a big enough issue that I switched to Tape for now.

novemberborn commented 3 years ago

@cakoose sounds like a use case t.try() is perfect for. If you want to discuss further let's move that conversation to https://github.com/avajs/ava/discussions.