jest-circus beforeAll hook behaviour on failure

[njmaeff]

🐛 Bug Report

When you have a failing beforeAll hook, all remaining beforeAll hooks will still run. Contrast that to beforeEach which after failing will skip the rest of the beforeEach hooks and skip the test.

To Reproduce

Steps to reproduce the behavior:

Expected behavior

I think after the first beforeAll failure, every other beforeAll hook and child test should be skipped / failed.

Link to repl or repo (highly encouraged)

import wrap from 'jest-snapshot-serializer-raw';
import {runTest} from 'jest-circus/src/__mocks__/testUtils';

test('a failing beforeAll will not skip other beforeAll hooks', () => {
    const {stdout} = runTest(`
    describe('test suite beforeAll', () => {

    beforeAll(() => {
        console.log('beforeAll 1 runs')
        throw new Error('beforeAll 1');
    });

    beforeAll(() => {
        console.log('beforeAll 2 runs')
        throw new Error('beforeAll 2');
    });

    test('should not run', () => {
        console.log('the test ran')
    });

});
  `);

    expect(wrap(stdout)).toMatchSnapshot();
});

// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`a failing beforeAll will not skip other beforeAll hooks 1`] = `
Object {
  Symbol(jest-snapshot-serializer-raw): "start_describe_definition: test suite beforeAll
add_hook: beforeAll
add_hook: beforeAll
add_test: should not run
finish_describe_definition: test suite beforeAll
run_start
run_describe_start: ROOT_DESCRIBE_BLOCK
run_describe_start: test suite beforeAll
hook_start: beforeAll
beforeAll 1 runs
hook_failure: beforeAll
hook_start: beforeAll
beforeAll 2 runs
hook_failure: beforeAll
test_start: should not run
test_fn_start: should not run
test_done: should not run
run_describe_finish: test suite beforeAll
run_describe_finish: ROOT_DESCRIBE_BLOCK
run_finish

unhandledErrors: 0",
}
`;

envinfo

[fider-apptimia]

Imo if beforeAll hook fail it should skip all related tests and afterAll hook should be run in any case (to ensure clean up is done, because beforeAll hook may initialize something partially)

[njmaeff]

@fider-apptimia The current behaviour for jest circus is to mark the test as error when a beforeAll hook fails. The test will not run but the rest of the beforeAll, afterEach, and afterAll hooks will still run. I feel like once the first beforeAll fails, the rest of the beforeAll hooks should skip and then afterAll hooks immediately run. There is an open issue to fix this in the jest jasmine runner. I didn't see any issue specifically related to circus for this behaviour.

[fider-apptimia]

@njmaeff imo issue here is that tests will be executed even if beforeAll throw.

[SimenB]

Tests should definitely not run if before* fails. Whether or not after* should run is probably a separate discussion

[thernstig]

afterEach() is also run in case beforeAll() fails. Is that a separate issue?

[github-actions[bot]]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[joshuapinter]

Stale? Has this been fixed. I think this is still an issue and should be fixed. I think @SimenB's comment summarized it perfectly:

Tests should definitely not run if before fails. Whether or not after should run is probably a separate discussion

[github-actions[bot]]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[joshuapinter]

Still not fixed as far as I'm aware of so we should keep it open.

[madcapnmckay]

I noticed that if I throw in beforeAll I still see the circus events for test_start, test_started & test_fn_start which i find strange since they occur after the hook_failure event, and I confirmed that the test function itself does not actually run.