search

avajs / ava

Node.js test runner that lets you develop with confidence 🚀
MIT License
20.74k stars 1.41k forks source link

Fail test even after it has passed #1330

Open novemberborn opened 7 years ago

novemberborn commented 7 years ago

09b23e0d5fd00d48f6b5b1f9c43641faa00f1d1d made it an error if an assertion is added after a test has finished. This can occur if the test finished earlier than the user intended. Once a test has passed though we cannot update its state in the reporters.

The feedback in 09b23e0d5fd00d48f6b5b1f9c43641faa00f1d1d could only be provided to users if the test was waiting for a pending assertion. That behavior itself is confusing, which means that when #1327 is fixed we won't be able to provide any feedback at all.

If a test has passed, but then later an assertion is added before other tests have finished, we should change the test state to a failure and update the reporters. The mini reporter should work well, since it hides previously passing tests. The verbose reporter would have a duplicate entry for the test but perhaps that's OK. I'm not sure whether it's acceptable to add a duplicate entry in the TAP reporter. The alternative is buffering the output for each test file which is not ideal.

sindresorhus commented 7 years ago

The verbose reporter would have a duplicate entry for the test but perhaps that's OK.

That's ok.

novemberborn commented 7 years ago

This also applies to callback tests ending more than once.

novemberborn commented 7 years ago

Note that this will still be best-effort. The test worker will exit once all tests have completed, even if there is a setTimeout() scheduled which double-ends a callback test, or makes an extraneous assertion.

novemberborn commented 6 years ago

With https://github.com/avajs/ava/pull/1776, we don't write test results until the test file has finished executing. So we'll be able to fail a test even if it seemed to pass at first.