metarhia / metatests

Extremely simple to use test framework and runner for Metarhia technology stack 🧪
https://metarhia.com
Other
18 stars 4 forks source link

Report on the test.rejects failure not working #278

Open KLarpen opened 11 months ago

KLarpen commented 11 months ago

Is your feature request related to a problem? Please describe.

I had encounter an issue when trying to look intended test fail results of Impress. Way to reproduce:

  1. Open impress folder in terminal.
  2. Open editor with file test/procedure.js
  3. In the assertion at lines 137-140 of the test metatests.testAsync('lib/procedure timeout'
  await test.rejects(
    async () => procedure.invoke({}, { waitTime: 150 }),
    new Error('Timeout of 100ms reached'),
  );

change the value of waitTime to 50. So the assertion must failed due to test configuration.

  1. Run npm t
  2. Instead of failed test report there is an error from reporting logic of metatest package itself (more specifically from one of its dependency tap-mocha-reporter)
    
    Node v20.9.0 (v8 11.3.244.8-node.16):
    Certificate request self-signature ok
    subject=CN = localhost
    lib/application ..................................... 21/21
    lib/deps .............................................. 8/8
    lib/procedure ....................................... 24/24
    lib/procedure validate ................................ 4/4
    schemas/contracts ..................................... 1/1
    lib/api load ........................................ 22/22
    lib/place ........................................... 18/18
    schemas/config ........................................ 4/4
    lib/bus ............................................. 11/11
    lib/cert .............................................. 4/4
    lib/code ............................................ 11/11
    lib/procedure timeout ../node_modules/tap-mocha-reporter/lib/runner.js:293
    return stack.trim().split('\n').map(function (line) {
                 ^

TypeError: stack.trim is not a function at reviveStack (/node_modules/tap-mocha-reporter/lib/runner.js:293:18) at getError (/node_modules/tap-mocha-reporter/lib/runner.js:332:87) at emitTest (/node_modules/tap-mocha-reporter/lib/runner.js:280:17) at Parser. (/node_modules/tap-mocha-reporter/lib/runner.js:219:5) at Parser.emit (node:events:514:28) at Parser.emit (node:domain:488:12) at Parser.emit (/node_modules/minipass/index.js:483:23) at Parser.emitAssert (/node_modules/tap-parser/index.js:864:10) at Parser.emitResult (/node_modules/tap-parser/index.js:733:10) at Parser.plan (/node_modules/tap-parser/index.js:417:10)


6. The error not only failing the report but breaking execution of subsequent test cases, e.g. 'lib/procedure validate async'. 

## Describe the solution you'd like

Introduce better reporters instead of tap-mocha-reporter as it was described in #265  

## Describe alternatives you've considered

Please propose temporary workaround. I had tried to investigate `function reviveStack` in `/node_modules/tap-mocha-reporter/lib/runner.js:293` by adding `console.debug({ stack, stackFile: stack.file })`. It seems that contract of `stack` that the function receives has been changed and even not stable to rely on: two internal subsequent calls resulting in string and object 

```sh
{
  stack: 'async ImperativeTest.func (/test/procedure.js:137:3)\n',
  stackFile: undefined
}
{
  stack: { file: '/test/procedure.js' },
  stackFile: '/test/procedure.js'
}

Unfortunately still had not found the temporary solution.

Additional context

OS: macOS 14.1.2 Node: 20.9.0, 18.18.2 Impress: 3.0.13 Metatests: 0.8.2

KLarpen commented 11 months ago

I had found temporary workaround.

  1. Open file /node_modules/tap-mocha-reporter/lib/runner.js
  2. Locate function reviveStack at line 286
  function reviveStack (stack) {
    if (!stack)
      return null

    return stack.trim().split('\n').map(function (line) {
      return '    at ' + line
    }).join('\n')
  }
  1. Change to
  function reviveStack (stack) {
    if (!stack)
      return null

    // TypeError: stack.trim is not a function
    // return stack.trim().split('\n').map(function (line) {
    //   return '    at ' + line
    // }).join('\n')
    // Workaround
    return stack.toString();
  }

Result of running npm t with the same setup as described above

Node v20.9.0 (v8 11.3.244.8-node.16):
Certificate request self-signature ok
subject=CN = localhost
lib/application ..................................... 21/21
lib/deps .............................................. 8/8
lib/procedure ....................................... 24/24
lib/procedure validate ................................ 4/4
schemas/contracts ..................................... 1/1
lib/api load ........................................ 22/22
lib/place ........................................... 18/18
schemas/config ........................................ 4/4
lib/bus ............................................. 11/11
lib/cert .............................................. 4/4
lib/code ............................................ 11/11
lib/static load ..................................... 11/11
lib/procedure timeout ................................. 0/2
  not ok rejects
    --- wanted                  
    +++ found                   
    -[null]                     
    +"success"                  
    message: expected to be rejected, but was resolved
    severity: fail
    type: rejects
    at:
      file: /test/procedure.js
    stack: >
      async ImperativeTest.func
      (/test/procedure.js:137:3)

  not ok fail
    --- wanted                  
    +++ found                   
    -[null]                     
    +"success"                  
    message: Promise rejection
    severity: fail
    type: fail
    at:
      file: /test/procedure.js
    stack: ""

lib/procedure validate async .......................... 4/4
FAILED test 14
Failed 1/15 tests, 93.33% okay
tshemsedinov commented 11 months ago

FYI @lundibundi

lundibundi commented 11 months ago

Fix in https://github.com/tapjs/tap-mocha-reporter/pull/79.

We can downgrade to tap-mocha-reporter@5.0.3 if that will be an issue for long.