qunitjs / js-reporters

📋 Common Reporter Interface (CRI) for JavaScript testing frameworks.
MIT License
59 stars 18 forks source link

Skipped tests reported incorrectly under Mocha 8.0+ #116

Closed Krinkle closed 3 years ago

Krinkle commented 3 years ago

The result of npm run test-integration with Mocha 8.0 and later is as follows:

log/failed--mocha--8.0.0.log ``` not ok 106 Adapters integration > Mocha adapter > Event "testEnd" runtime property --- message: "failed" severity: failed actual : "undefined" expected: "number" stack: at collectedData.forEach.value (/js-reporters/test/integration/adapters.js:185:20) at Array.forEach () at Object.assert (/js-reporters/test/integration/adapters.js:183:23) ... --- message: "failed" severity: failed actual : "undefined" expected: "number" stack: at collectedData.forEach.value (/js-reporters/test/integration/adapters.js:185:20) at Array.forEach () at Object.assert (/js-reporters/test/integration/adapters.js:183:23) ... ok 107 Adapters integration > Mocha adapter > Event "testEnd" errors property ok 108 Adapters integration > Mocha adapter > Event "testEnd" assertions property ok 109 Adapters integration > Mocha adapter > global suite starts ok 110 Adapters integration > Mocha adapter > global test starts ok 111 Adapters integration > Mocha adapter > global test ends ok 112 Adapters integration > Mocha adapter > suite with one passing test starts ok 113 Adapters integration > Mocha adapter > passing test starts ok 114 Adapters integration > Mocha adapter > passing test ends ok 115 Adapters integration > Mocha adapter > suite with one passing test ends ok 116 Adapters integration > Mocha adapter > suite with one skipped test starts ok 117 Adapters integration > Mocha adapter > skipped test starts not ok 118 Adapters integration > Mocha adapter > skipped test ends --- message: "failed" severity: failed actual : { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] } expected: { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "skipped", "errors": [], "assertions": [] } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... not ok 119 Adapters integration > Mocha adapter > suite with one skipped test ends --- message: "failed" severity: failed actual : "failed" expected: "skipped" stack: at Object.assert (/js-reporters/test/integration/adapters.js:295:20) ... --- message: "failed" severity: failed actual : { "passed": 0, "failed": 0, "skipped": 0, "todo": 0, "total": 1 } expected: { "passed": 0, "failed": 0, "skipped": 1, "todo": 0, "total": 1 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:305:20) ... --- message: "failed" severity: failed actual : { "name": "suite with skipped test", "fullName": [ "suite with skipped test" ], "tests": [ { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 0, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 } expected: { "name": "suite with skipped test", "fullName": [ "suite with skipped test" ], "tests": [ { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "skipped", "errors": [], "assertions": [] } ], "childSuites": [], "status": "skipped", "testCounts": { "passed": 0, "failed": 0, "skipped": 1, "todo": 0, "total": 1 } } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... ok 120 Adapters integration > Mocha adapter > suite with one failing tests ok 121 Adapters integration > Mocha adapter > failing test starts ok 122 Adapters integration > Mocha adapter > failing test ends ok 123 Adapters integration > Mocha adapter > suite with one failing test ends ok 124 Adapters integration > Mocha adapter > suite with multiple tests starts ok 125 Adapters integration > Mocha adapter > passing test starts ok 126 Adapters integration > Mocha adapter > passing test ends ok 127 Adapters integration > Mocha adapter > skipped test starts not ok 128 Adapters integration > Mocha adapter > skipped test ends --- message: "failed" severity: failed actual : { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] } expected: { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "skipped", "errors": [], "assertions": [] } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... ok 129 Adapters integration > Mocha adapter > failing test starts ok 130 Adapters integration > Mocha adapter > failing test ends not ok 131 Adapters integration > Mocha adapter > suite with multiple tests ends --- message: "failed" severity: failed actual : { "passed": 1, "failed": 1, "skipped": 0, "todo": 0, "total": 3 } expected: { "passed": 1, "failed": 1, "skipped": 1, "todo": 0, "total": 3 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:305:20) ... --- message: "failed" severity: failed actual : { "name": "suite with tests", "fullName": [ "suite with tests" ], "tests": [ { "name": "should pass", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] }, { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] }, { "name": "should fail", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 1, "failed": 1, "skipped": 0, "todo": 0, "total": 3 }, "runtime": 0 } expected: { "name": "suite with tests", "fullName": [ "suite with tests" ], "tests": [ { "name": "should pass", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] }, { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "skipped", "errors": [], "assertions": [] }, { "name": "should fail", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 1, "failed": 1, "skipped": 1, "todo": 0, "total": 3 }, "runtime": 0 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... ok 132 Adapters integration > Mocha adapter > outter suite starts ok 133 Adapters integration > Mocha adapter > outter test starts ok 134 Adapters integration > Mocha adapter > outter test ends ok 135 Adapters integration > Mocha adapter > inner suite starts ok 136 Adapters integration > Mocha adapter > inner test starts ok 137 Adapters integration > Mocha adapter > inner test ends ok 138 Adapters integration > Mocha adapter > inner suite ends ok 139 Adapters integration > Mocha adapter > outter suite ends not ok 140 Adapters integration > Mocha adapter > global suite ends --- message: "failed" severity: failed actual : { "passed": 5, "failed": 2, "skipped": 0, "todo": 0, "total": 9 } expected: { "passed": 5, "failed": 2, "skipped": 2, "todo": 0, "total": 9 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:305:20) ... --- message: "failed" severity: failed actual : { "fullName": [], "tests": [ { "name": "global test", "fullName": [ "global test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [ { "name": "suite with passing test", "fullName": [ "suite with passing test" ], "tests": [ { "name": "should pass", "suiteName": "suite with passing test", "fullName": [ "suite with passing test", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [], "status": "passed", "testCounts": { "passed": 1, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with skipped test", "fullName": [ "suite with skipped test" ], "tests": [ { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 0, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with failing test", "fullName": [ "suite with failing test" ], "tests": [ { "name": "should fail", "suiteName": "suite with failing test", "fullName": [ "suite with failing test", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 0, "failed": 1, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with tests", "fullName": [ "suite with tests" ], "tests": [ { "name": "should pass", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] }, { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] }, { "name": "should fail", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 1, "failed": 1, "skipped": 0, "todo": 0, "total": 3 }, "runtime": 0 }, { "name": "outter suite", "fullName": [ "outter suite" ], "tests": [ { "name": "outter test", "suiteName": "outter suite", "fullName": [ "outter suite", "outter test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [ { "name": "inner suite", "fullName": [ "outter suite", "inner suite" ], "tests": [ { "name": "inner test", "suiteName": "inner suite", "fullName": [ "outter suite", "inner suite", "inner test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [], "status": "passed", "testCounts": { "passed": 1, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 } ], "status": "passed", "testCounts": { "passed": 2, "failed": 0, "skipped": 0, "todo": 0, "total": 2 }, "runtime": 0 } ], "status": "failed", "testCounts": { "passed": 5, "failed": 2, "skipped": 0, "todo": 0, "total": 9 }, "runtime": 0 } expected: { "fullName": [], "tests": [ { "name": "global test", "fullName": [ "global test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [ { "name": "suite with passing test", "fullName": [ "suite with passing test" ], "tests": [ { "name": "should pass", "suiteName": "suite with passing test", "fullName": [ "suite with passing test", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [], "status": "passed", "testCounts": { "passed": 1, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with skipped test", "fullName": [ "suite with skipped test" ], "tests": [ { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "skipped", "errors": [], "assertions": [] } ], "childSuites": [], "status": "skipped", "testCounts": { "passed": 0, "failed": 0, "skipped": 1, "todo": 0, "total": 1 } }, { "name": "suite with failing test", "fullName": [ "suite with failing test" ], "tests": [ { "name": "should fail", "suiteName": "suite with failing test", "fullName": [ "suite with failing test", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 0, "failed": 1, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with tests", "fullName": [ "suite with tests" ], "tests": [ { "name": "should pass", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] }, { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "skipped", "errors": [], "assertions": [] }, { "name": "should fail", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 1, "failed": 1, "skipped": 1, "todo": 0, "total": 3 }, "runtime": 0 }, { "name": "outter suite", "fullName": [ "outter suite" ], "tests": [ { "name": "outter test", "suiteName": "outter suite", "fullName": [ "outter suite", "outter test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [ { "name": "inner suite", "fullName": [ "outter suite", "inner suite" ], "tests": [ { "name": "inner test", "suiteName": "inner suite", "fullName": [ "outter suite", "inner suite", "inner test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [], "status": "passed", "testCounts": { "passed": 1, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 } ], "status": "passed", "testCounts": { "passed": 2, "failed": 0, "skipped": 0, "todo": 0, "total": 2 }, "runtime": 0 } ], "status": "failed", "testCounts": { "passed": 5, "failed": 2, "skipped": 2, "todo": 0, "total": 9 }, "runtime": 0 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... 1..140 # pass 134 # skip 0 # todo 0 # fail 6 ```
log/failed--mocha--8.1.3.log ``` not ok 106 Adapters integration > Mocha adapter > Event "testEnd" runtime property --- message: "failed" severity: failed actual : "undefined" expected: "number" stack: at collectedData.forEach.value (/js-reporters/test/integration/adapters.js:185:20) at Array.forEach () at Object.assert (/js-reporters/test/integration/adapters.js:183:23) ... --- message: "failed" severity: failed actual : "undefined" expected: "number" stack: at collectedData.forEach.value (/js-reporters/test/integration/adapters.js:185:20) at Array.forEach () at Object.assert (/js-reporters/test/integration/adapters.js:183:23) ... ok 107 Adapters integration > Mocha adapter > Event "testEnd" errors property ok 108 Adapters integration > Mocha adapter > Event "testEnd" assertions property ok 109 Adapters integration > Mocha adapter > global suite starts ok 110 Adapters integration > Mocha adapter > global test starts ok 111 Adapters integration > Mocha adapter > global test ends ok 112 Adapters integration > Mocha adapter > suite with one passing test starts ok 113 Adapters integration > Mocha adapter > passing test starts ok 114 Adapters integration > Mocha adapter > passing test ends ok 115 Adapters integration > Mocha adapter > suite with one passing test ends ok 116 Adapters integration > Mocha adapter > suite with one skipped test starts ok 117 Adapters integration > Mocha adapter > skipped test starts not ok 118 Adapters integration > Mocha adapter > skipped test ends --- message: "failed" severity: failed actual : { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] } expected: { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "skipped", "errors": [], "assertions": [] } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... not ok 119 Adapters integration > Mocha adapter > suite with one skipped test ends --- message: "failed" severity: failed actual : "failed" expected: "skipped" stack: at Object.assert (/js-reporters/test/integration/adapters.js:295:20) ... --- message: "failed" severity: failed actual : { "passed": 0, "failed": 0, "skipped": 0, "todo": 0, "total": 1 } expected: { "passed": 0, "failed": 0, "skipped": 1, "todo": 0, "total": 1 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:305:20) ... --- message: "failed" severity: failed actual : { "name": "suite with skipped test", "fullName": [ "suite with skipped test" ], "tests": [ { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 0, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 } expected: { "name": "suite with skipped test", "fullName": [ "suite with skipped test" ], "tests": [ { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "skipped", "errors": [], "assertions": [] } ], "childSuites": [], "status": "skipped", "testCounts": { "passed": 0, "failed": 0, "skipped": 1, "todo": 0, "total": 1 } } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... ok 120 Adapters integration > Mocha adapter > suite with one failing tests ok 121 Adapters integration > Mocha adapter > failing test starts ok 122 Adapters integration > Mocha adapter > failing test ends ok 123 Adapters integration > Mocha adapter > suite with one failing test ends ok 124 Adapters integration > Mocha adapter > suite with multiple tests starts ok 125 Adapters integration > Mocha adapter > passing test starts ok 126 Adapters integration > Mocha adapter > passing test ends ok 127 Adapters integration > Mocha adapter > skipped test starts not ok 128 Adapters integration > Mocha adapter > skipped test ends --- message: "failed" severity: failed actual : { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] } expected: { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "skipped", "errors": [], "assertions": [] } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... ok 129 Adapters integration > Mocha adapter > failing test starts ok 130 Adapters integration > Mocha adapter > failing test ends not ok 131 Adapters integration > Mocha adapter > suite with multiple tests ends --- message: "failed" severity: failed actual : { "passed": 1, "failed": 1, "skipped": 0, "todo": 0, "total": 3 } expected: { "passed": 1, "failed": 1, "skipped": 1, "todo": 0, "total": 3 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:305:20) ... --- message: "failed" severity: failed actual : { "name": "suite with tests", "fullName": [ "suite with tests" ], "tests": [ { "name": "should pass", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] }, { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] }, { "name": "should fail", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 1, "failed": 1, "skipped": 0, "todo": 0, "total": 3 }, "runtime": 0 } expected: { "name": "suite with tests", "fullName": [ "suite with tests" ], "tests": [ { "name": "should pass", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] }, { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "skipped", "errors": [], "assertions": [] }, { "name": "should fail", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 1, "failed": 1, "skipped": 1, "todo": 0, "total": 3 }, "runtime": 0 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... ok 132 Adapters integration > Mocha adapter > outter suite starts ok 133 Adapters integration > Mocha adapter > outter test starts ok 134 Adapters integration > Mocha adapter > outter test ends ok 135 Adapters integration > Mocha adapter > inner suite starts ok 136 Adapters integration > Mocha adapter > inner test starts ok 137 Adapters integration > Mocha adapter > inner test ends ok 138 Adapters integration > Mocha adapter > inner suite ends ok 139 Adapters integration > Mocha adapter > outter suite ends not ok 140 Adapters integration > Mocha adapter > global suite ends --- message: "failed" severity: failed actual : { "passed": 5, "failed": 2, "skipped": 0, "todo": 0, "total": 9 } expected: { "passed": 5, "failed": 2, "skipped": 2, "todo": 0, "total": 9 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:305:20) ... --- message: "failed" severity: failed actual : { "fullName": [], "tests": [ { "name": "global test", "fullName": [ "global test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [ { "name": "suite with passing test", "fullName": [ "suite with passing test" ], "tests": [ { "name": "should pass", "suiteName": "suite with passing test", "fullName": [ "suite with passing test", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [], "status": "passed", "testCounts": { "passed": 1, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with skipped test", "fullName": [ "suite with skipped test" ], "tests": [ { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 0, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with failing test", "fullName": [ "suite with failing test" ], "tests": [ { "name": "should fail", "suiteName": "suite with failing test", "fullName": [ "suite with failing test", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 0, "failed": 1, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with tests", "fullName": [ "suite with tests" ], "tests": [ { "name": "should pass", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] }, { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "pending", "runtime": 0, "errors": [], "assertions": [] }, { "name": "should fail", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 1, "failed": 1, "skipped": 0, "todo": 0, "total": 3 }, "runtime": 0 }, { "name": "outter suite", "fullName": [ "outter suite" ], "tests": [ { "name": "outter test", "suiteName": "outter suite", "fullName": [ "outter suite", "outter test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [ { "name": "inner suite", "fullName": [ "outter suite", "inner suite" ], "tests": [ { "name": "inner test", "suiteName": "inner suite", "fullName": [ "outter suite", "inner suite", "inner test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [], "status": "passed", "testCounts": { "passed": 1, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 } ], "status": "passed", "testCounts": { "passed": 2, "failed": 0, "skipped": 0, "todo": 0, "total": 2 }, "runtime": 0 } ], "status": "failed", "testCounts": { "passed": 5, "failed": 2, "skipped": 0, "todo": 0, "total": 9 }, "runtime": 0 } expected: { "fullName": [], "tests": [ { "name": "global test", "fullName": [ "global test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [ { "name": "suite with passing test", "fullName": [ "suite with passing test" ], "tests": [ { "name": "should pass", "suiteName": "suite with passing test", "fullName": [ "suite with passing test", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [], "status": "passed", "testCounts": { "passed": 1, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with skipped test", "fullName": [ "suite with skipped test" ], "tests": [ { "name": "should skip", "suiteName": "suite with skipped test", "fullName": [ "suite with skipped test", "should skip" ], "status": "skipped", "errors": [], "assertions": [] } ], "childSuites": [], "status": "skipped", "testCounts": { "passed": 0, "failed": 0, "skipped": 1, "todo": 0, "total": 1 } }, { "name": "suite with failing test", "fullName": [ "suite with failing test" ], "tests": [ { "name": "should fail", "suiteName": "suite with failing test", "fullName": [ "suite with failing test", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 0, "failed": 1, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 }, { "name": "suite with tests", "fullName": [ "suite with tests" ], "tests": [ { "name": "should pass", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should pass" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] }, { "name": "should skip", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should skip" ], "status": "skipped", "errors": [], "assertions": [] }, { "name": "should fail", "suiteName": "suite with tests", "fullName": [ "suite with tests", "should fail" ], "status": "failed", "runtime": 0, "errors": [ { "passed": false } ], "assertions": [ { "passed": false } ] } ], "childSuites": [], "status": "failed", "testCounts": { "passed": 1, "failed": 1, "skipped": 1, "todo": 0, "total": 3 }, "runtime": 0 }, { "name": "outter suite", "fullName": [ "outter suite" ], "tests": [ { "name": "outter test", "suiteName": "outter suite", "fullName": [ "outter suite", "outter test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [ { "name": "inner suite", "fullName": [ "outter suite", "inner suite" ], "tests": [ { "name": "inner test", "suiteName": "inner suite", "fullName": [ "outter suite", "inner suite", "inner test" ], "status": "passed", "runtime": 0, "errors": [], "assertions": [ { "passed": true } ] } ], "childSuites": [], "status": "passed", "testCounts": { "passed": 1, "failed": 0, "skipped": 0, "todo": 0, "total": 1 }, "runtime": 0 } ], "status": "passed", "testCounts": { "passed": 2, "failed": 0, "skipped": 0, "todo": 0, "total": 2 }, "runtime": 0 } ], "status": "failed", "testCounts": { "passed": 5, "failed": 2, "skipped": 2, "todo": 0, "total": 9 }, "runtime": 0 } stack: at Object.assert (/js-reporters/test/integration/adapters.js:315:18) ... 1..140 # pass 134 # skip 0 # todo 0 # fail 6 ```

See also:

Krinkle commented 3 years ago

Skipped tests used to have an undefined state, but as of Mocha 8.0 they have the "pending" state. Perhaps we can safely map that to "skipped"? Maybe not, since it sounds like it would be used for tests that will run but haven't yet?

/cc @js-reporters/mocha Would welcome your recomendation on this matter :)

Krinkle commented 3 years ago

Tracked down to https://github.com/mochajs/mocha/commit/8236ffd7494 (https://github.com/mochajs/mocha/pull/4223), which introduced a new state STATE_PENDING ("pending") and assigned it to the .state property which was previously left undefined for this case.

This wasn't mentioned in the changelog for Mocha 8.0.0, but I guess that's fair since the public API has offered isPending() for many years which still works the same as before and actually uses a separate property .pending = true and has for many years, so it didn't even need updating for this change.