NodeJS Test Runner

[nathanjhood]

Really enjoying NodeJS Test Runner - so neat having these common test utils without any major dependencies or config.

The downside is that those factors are to be self-managed. It's a case of building simple simple, shallow tests that don't need tests of their own, versus scaling up some sort of test factory.

I'm in favour of keeping the tests as shallow as possible; yet I'd also like to use this opportunity to learn, so no need to be too shy.

Current issue: the test runner is working great in the pipeline, even reporting the failure that I forgot to create a .env file for the parseEnv function's tests. However, the test job passes with status code 0 - this is a false positive.

Output from last CI run, MacOS latest with Node 20:

Run yarn test
yarn run v1.22.22
$ ./test/run.ts
$ /Users/runner/work/esbuild-scripts/esbuild-scripts/node_modules/.bin/tsx ./test/run.ts
▶ parseCommand
  ▶ imports
    ✔ require (16.279792ms)
    ✔ import (17.096708ms)
    ✔ import <Promise> (0.672625ms)
    ✔ import (async) (0.772292ms)
  ▶ imports (35.787791ms)
▶ parseCommand (37.990792ms)
▶ parseArgV
  ▶ imports
    ✔ require (5.528292ms)
    ✔ import (14.361208ms)
    ✔ import <Promise> (3.347666ms)
    ✔ import (async) (0.39[11](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905273426/job/30263719195#step:5:12)25ms)
  ▶ imports (24.296292ms)
▶ parseArgV (26.164583ms)
▶ parseEnv
  ▶ imports
    ✔ require (3.431541ms)
    ✔ import (7.664416ms)
    ✔ import <Promise> (0.642084ms)
    ✔ import (async) (0.321708ms)
  ▶ imports ([12](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905273426/job/30263719195#step:5:13).592125ms)
  ▶ runs
    ✖ loads .env from cwd() (0.332334ms)
      Error: parseEnv failed
          at <anonymous> (/Users/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:22:11) {
        [cause]: Error: ENOENT: no such file or directory, open '/Users/runner/work/esbuild-scripts/esbuild-scripts/.env'
            at loadEnvFile (node:internal/process/per_thread:261:7)
            at Promise.catch.Error.cause (/Users/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:18:5)
            at new Promise (<anonymous>)
            at parseEnv (/Users/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:8:10)
            at TestContext.<anonymous> (/Users/runner/work/esbuild-scripts/esbuild-scripts/test/process/parseEnv.test.ts:87:7)
            at Test.runInAsyncScope (node:async_hooks:206:9)
            at Test.run (node:internal/test_runner/test:776:21)
            at Test.start (node:internal/test_runner/test:693:17)
            at node:internal/test_runner/test:1129:71
            at node:internal/per_context/primordials:487:82 {
          errno: -2,
          code: 'ENOENT',
          syscall: 'open',
          path: '/Users/runner/work/esbuild-scripts/esbuild-scripts/.env'
        }
      }

  ▶ runs (0.446666ms)
▶ parseEnv ([13](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905273426/job/30263719195#step:5:14).6125ms)
▶ cli
  ▶ imports
    ✔ require (5.913709ms)
    ✔ import (6.876583ms)
    ✔ import <Promise> (0.862542ms)
    ✔ import (async) (0.213209ms)
  ▶ imports ([14](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905273426/job/30263719195#step:5:15).419334ms)
▶ cli (14.859792ms)
ℹ tests 17
ℹ suites 9
ℹ pass [16](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905273426/job/30263719195#step:5:17)
ℹ fail 1
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 529.013125

✖ failing tests:

test at test/process/parseEnv.test.ts:2:2480
✖ loads .env from cwd() (0.332334ms)
  Error: parseEnv failed
      at <anonymous> (/Users/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:22:11) {
    [cause]: Error: ENOENT: no such file or directory, open '/Users/runner/work/esbuild-scripts/esbuild-scripts/.env'
        at loadEnvFile (node:internal/process/per_thread:261:7)
        at Promise.catch.Error.cause (/Users/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:18:5)
        at new Promise (<anonymous>)
        at parseEnv (/Users/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:8:10)
        at TestContext.<anonymous> (/Users/runner/work/esbuild-scripts/esbuild-scripts/test/process/parseEnv.test.ts:87:7)
        at Test.runInAsyncScope (node:async_hooks:206:9)
        at Test.run (node:internal/test_runner/test:776:21)
        at Test.start (node:internal/test_runner/test:693:[17](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905273426/job/30263719195#step:5:18))
        at node:internal/test_runner/test:1129:71
        at node:internal/per_context/primordials:487:82 {
      errno: -2,
      code: 'ENOENT',
      syscall: 'open',
      path: '/Users/runner/work/esbuild-scripts/esbuild-scripts/.env'
    }
  }
process 3382 exiting with code 0
process 3382 exited with code 0
Done in 0.96s.

We get the green 'tick' and everything and the job proceeds to the build stage. This test actually failed, and thus the test workflow should not have been considered successful and exited != 0 (and, the following build job should have been cancelled, saving us pipeline minutes building a broken app)...

I think I've mistakenly over-ridden the correct exit procedure, or possibly need to refactor parseEnv to throw/fail differently.

I also made a unit test which imports cli() and checks it, but this fails (as it should...) since there was no argument passed to it. Consider that the method is supposed to be invoked with arguments over stdin, so the test may need to use (or somehow, mock?) child_process.spawn... That, or use some kind of testing layer which is invoked over stdin and calls/pipes/streams/wraps the CLI app somehow. Another idea would be simply to inline something like process.argV.push('build') right before running the assertion test.

An additional issue is that I'm not seeing NodeJS Test Runner printing to stdout on the Windows run - there is nothing to suggest that the tests actually did run at all on that workflow...

Ultimately, this is precisely why I've started this project - getting to the bottom of "this kind of thing" and learning how to get it right.

[nathanjhood]

Added an event listener to the test:fail event in test/setupTests/setup().

The Ubuntu run, being the fastest, now "fails successfully" - and prevents not just the build step, but also all the other GitHub workflows as expected.

The output is very verbose here, due to double-logging of the test failure. My hope is that Windows terminal reporting might work by using the event hooks explicitly for (all?) test runner/reporter logging.

Run yarn test
yarn run v1.22.22
$ ./test/run.ts
$ /home/runner/work/esbuild-scripts/esbuild-scripts/node_modules/.bin/tsx ./test/run.ts
▶ parseCommand
  ▶ imports
    ✔ require ([11](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905944978/job/30266041390#step:5:12).461562ms)
    ✔ import (15.852999ms)
    ✔ import <Promise> (2.645078ms)
    ✔ import (async) (0.774168ms)
  ▶ imports (31.91659ms)
▶ parseCommand (33.211403ms)
▶ parseArgV
  ▶ imports
    ✔ require (7.855656ms)
    ✔ import (15.209907ms)
    ✔ import <Promise> (3.397051ms)
    ✔ import (async) (0.778506ms)
  ▶ imports (28.435747ms)
▶ parseArgV (29.63[12](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905944978/job/30266041390#step:5:13)53ms)
▶ parseEnv
  ▶ imports
    ✔ require (7.949671ms)
    ✔ import ([13](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905944978/job/30266041390#step:5:14).840235ms)
    ✔ import <Promise> (1.569958ms)
    ✔ import (async) (0.688658ms)
  ▶ imports (25.764883ms)
{
  ▶ runs
  name: 'loads .env from cwd()',
    ✖ loads .env from cwd() (0.888011ms)
  nesting: 2,
      Error: parseEnv failed
  testNumber: 1,
  details: {
          at <anonymous> (/home/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:22:11) {
    duration_ms: 0.888011,
    error: Error [ERR_TEST_FAILURE]: parseEnv failed
        at async Promise.all (index 0) {
      code: 'ERR_TEST_FAILURE',
      failureType: 'testCodeFailure',
Error: ause: [Error]
    }
  },
  line: 2,
  column: 2480,
  file: '/home/runner/work/esbuild-scripts/esbuild-scripts/test/process/parseEnv.test.ts'
}
{
  name: 'runs',
        [cause]: Error: ENOENT: no such file or directory, open '/home/runner/work/esbuild-scripts/esbuild-scripts/.env'
            at loadEnvFile (node:internal/process/per_thread:261:7)
            at Promise.catch.Error.cause (/home/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:18:5)
            at new Promise (<anonymous>)
            at parseEnv (/home/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:8:10)
            at TestContext.<anonymous> (/home/runner/work/esbuild-scripts/esbuild-scripts/test/process/parseEnv.test.ts:87:7)
            at Test.runInAsyncScope (node:async_hooks:206:9)
            at Test.run (node:internal/test_runner/test:776:21)
            at Test.start (node:internal/test_runner/test:693:17)
            at node:internal/test_runner/test:1129:71
            at node:internal/per_context/primordials:487:82 {
          errno: -2,
          code: 'ENOENT',
          syscall: 'open',
          path: '/home/runner/work/esbuild-scripts/esbuild-scripts/.env'
        }
      }

  nesting: 1,
  testNumber: 2,
  details: {
    duration_ms: 1.197009,
    type: 'suite',
    error: [Error [ERR_TEST_FAILURE]: 1 subtest failed] {
      code: 'ERR_TEST_FAILURE',
      failureType: 'subtestsFailed',
      cause: '1 subtest failed'
    }
  },
  line: 2,
  column: 2393,
  file: '/home/runner/work/esbuild-scripts/esbuild-scripts/test/process/parseEnv.test.ts'
}
{
  name: 'parseEnv',
  nesting: 0,
  testNumber: 3,
  details: {
    duration_ms: 28.6[14](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905944978/job/30266041390#step:5:15)063,
    type: 'suite',
    error: [Error [ERR_TEST_FAILURE]: 1 subtest failed] {
      code: 'ERR_TEST_FAILURE',
      failureType: 'subtestsFailed',
      cause: '1 subtest failed'
    }
  },
  line: 2,
  column: 855,
  file: '/home/runner/work/esbuild-scripts/esbuild-scripts/test/process/parseEnv.test.ts'
}
  ▶ runs (1.197009ms)
▶ parseEnv (28.614063ms)
▶ cli
  ▶ imports
    ✔ require (11.029622ms)
    ✔ import (13.838273ms)
    ✔ import <Promise> (1.703017ms)
    ✔ import (async) (0.463196ms)
  ▶ imports (28.193407ms)
▶ cli (29.053426ms)
ℹ tests 17
ℹ suites 9
ℹ pass [16](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905944978/job/30266041390#step:5:17)
ℹ fail 1
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 677.885723

✖ failing tests:

test at test/process/parseEnv.test.ts:2:2480
✖ loads .env from cwd() (0.888011ms)
  Error: parseEnv failed
      at <anonymous> (/home/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:22:11) {
    [cause]: Error: ENOENT: no such file or directory, open '/home/runner/work/esbuild-scripts/esbuild-scripts/.env'
        at loadEnvFile (node:internal/process/per_thread:261:7)
        at Promise.catch.Error.cause (/home/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:18:5)
        at new Promise (<anonymous>)
        at parseEnv (/home/runner/work/esbuild-scripts/esbuild-scripts/src/process/parseEnv.ts:8:10)
        at TestContext.<anonymous> (/home/runner/work/esbuild-scripts/esbuild-scripts/test/process/parseEnv.test.ts:87:7)
        at Test.runInAsyncScope (node:async_hooks:206:9)
        at Test.run (node:internal/test_runner/test:776:21)
        at Test.start (node:internal/test_runner/test:693:[17](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905944978/job/30266041390#step:5:18))
        at node:internal/test_runner/test:1129:71
        at node:internal/per_context/primordials:487:82 {
      errno: -2,
      code: 'ENOENT',
      syscall: 'open',
      path: '/home/runner/work/esbuild-scripts/esbuild-scripts/.env'
    }
  }
process [18](https://github.com/nathanjhood/esbuild-scripts/actions/runs/10905944978/job/30266041390#step:5:19)93 exiting with code 1
process 1893 exited with code 1
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Error: Process completed with exit code 1.

I added a .env.example - this can be cp'd in the pipeline to allow the test to now pass, since rooting out the false positive.

I will add a comment to the line which fixed this for future ref.

[nathanjhood]

The pipelines are passing now.

No Windows terminal reporting, though...

[nathanjhood]

Some fixes done:

• added cp .env.example .env to GitHub workflow file node.yml
• added a check to process.stdout.isTTY and set the test runner reporter to spec if true, otherwise use tap (actually the default behaviour)
• added some test suites to do basic 'run' checks on the process/parse* functions
• confirmed that Windows does appear to be running the tests (it is running slower since adding more), but is still not printing to stdout, possibly since we are using bash instead of powershell?
• noted that the cli function is more complicated to test, since it expects arguments, but this also presents the opportunity to test that it "fails as expected", which we can assert for

Next, I will write another process/parse*-type function, probably a simple one like parseCwd(), and attempt the following:

• write the test file first
• write the declaration file according to the findings and workings of the test(s)
• write the actual implementation file (*.ts) last

[nathanjhood]

Added:

• **/process/parseCwd test, declaration, and implementation files, including a parseCwdSync method

[nathanjhood]

The pipelines are passing now.

No Windows terminal reporting, though...

fixed - needed to call node/tsx explicitly in package.json for those scripts - these are dev-side commands, not exported to consumers, so I'm happy to make the concession. Some concern remains whether the issue will re-appear once consuming this package on Windows platforms, but I'm taking some confidence in react-scripts not seeming to have such an issue.

[nathanjhood]

testing-library/cli-testing-library might just be worth a look...

[nathanjhood]

Added a mock spy for the parse* functions like so:

test.describe('parseEnv() test suite', () => {
  const mock = test.mock;
  test.it('spies on our imported function', (ctx, done) => {
    const parseEnv = require('./path/to/parseEnv');
    const parseEnvSpy = ctx.mock.fn(parseEnv); // pass 'parseEnv' into 'mock.fn' to create a spy...
    const env = parseEnvSpy(global.process);
    ctx.assert.ok(env);
    ctx.assert.deepStrictEqual(parseEnvSpy.mock.callcount(), 1);
    return done();
  })
})

[nathanjhood]

Tried adding NodeJS v18 support, failed:

Run yarn test
yarn run v1.22.22
$ tsx test/run.ts
/home/runner/work/esbuild-scripts/esbuild-scripts/test/run.ts:28
    throw error;
    ^

TypeError: loadEnvFile is not a function

[nathanjhood]

Nothing really to add to this now; some good findings, though.

I'm looking forward to exploring the depths of NodeJS Test Runner in some more specific PR's in due course.