[v20.12.2 -> v20.13.0 regression] TestRunner: awaited variables no longer in scope

[sethvargo]

Version

v20.13.1

Platform

Darwin, but reproduces on Ubuntu too

Subsystem

No response

What steps will reproduce the bug?

The following test passes on v20.12.2 but fails on v20.13.1 (and v20.13.0):

// test.js
// node --test
import { test } from 'node:test';
import assert from 'node:assert';

test('outer', async (suite) => {
  let foo;

  suite.before(async () => {
    await new Promise((resolve) => setTimeout(resolve, 1000));
    foo = 'set value';
  });

  await suite.test('test', async () => {
    assert.ok(foo);
  });
});

Of note, the before function is awaited. In v20.12, the before properly blocks execution of tests until fully awaited. In v20.13, tests immediately begin executing, meaning the local variable is not set.

$ node -v
v20.12.2

$ node --test
▶ outer
  ✔ test (1004.618292ms)
▶ outer (1007.326292ms)

ℹ tests 2
ℹ suites 0
ℹ pass 2
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 1063.150791

$ node -v
v20.13.1

$ node --test
▶ outer
  ✖ test (6.006375ms)
    AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

      assert.ok(foo)

        at TestContext.<anonymous> (file:///Users/sethvargo/Desktop/foo/test.js:13:12)
        at Test.runInAsyncScope (node:async_hooks:206:9)
        at Test.run (node:internal/test_runner/test:656:25)
        at async TestContext.<anonymous> (file:///Users/sethvargo/Desktop/foo/test.js:12:3)
        at async Test.run (node:internal/test_runner/test:657:9)
        at async startSubtest (node:internal/test_runner/harness:235:3) {
      generatedMessage: true,
      code: 'ERR_ASSERTION',
      actual: undefined,
      expected: true,
      operator: '=='
    }

▶ outer (7.11475ms)
ℹ tests 2
ℹ suites 0
ℹ pass 0
ℹ fail 2
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 1049.811625

✖ failing tests:

test at file:/Users/sethvargo/Desktop/foo/test.js:12:15
✖ test (6.006375ms)
  AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

    assert.ok(foo)

      at TestContext.<anonymous> (file:///Users/sethvargo/Desktop/foo/test.js:13:12)
      at Test.runInAsyncScope (node:async_hooks:206:9)
      at Test.run (node:internal/test_runner/test:656:25)
      at async TestContext.<anonymous> (file:///Users/sethvargo/Desktop/foo/test.js:12:3)
      at async Test.run (node:internal/test_runner/test:657:9)
      at async startSubtest (node:internal/test_runner/harness:235:3) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: undefined,
    expected: true,
    operator: '=='
  }

and just for good measure, this does not exist in v22:

$ node -v
v22.2.0

$ node --test
▶ outer
  ✔ test (1007.872875ms)
▶ outer (1009.431875ms)
ℹ tests 2
ℹ suites 0
ℹ pass 2
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 1069.456334

How often does it reproduce? Is there a required condition?

This happens 100% of the time.

What is the expected behavior? Why is that the expected behavior?

I expected TestRunner API stability between v20.12 and v20.13.

What do you see instead?

Tests that previously passed in v20.12 are failing in v20.13.

Additional information

https://github.com/nodejs/node/pull/51909 looks suspicious, but I'm not an expert here.

[cjihrig]

This looks similar to https://github.com/nodejs/node/issues/52764. I believe that issue was caused by https://github.com/nodejs/node/pull/52003, and fixed by https://github.com/nodejs/node/pull/52791. If so, that fix was released in v22.2.0, and should be backported to v20 soon.

[sethvargo]

Is there a way to validate that? I confirmed v22 is working, so maybe that's enough?

[cjihrig]

I think that is the best way to validate it short of building Node 20 locally with that patch applied.