digitalbazaar / mocha-w3c-interop-reporter

Mocha W3C Interoperability Test Suite reporter
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Parallelize Reporter in v2 #16

Open aljones15 opened 2 years ago

aljones15 commented 2 years ago

Running this reporter in parallel has turned out to be difficult as the reporter relies on mutating mocha's test object with meta data the reporter uses. My proposal is to remove the meta-data and instead create a new javascript file required by the reporter and potentially mocha that contains information on how to format and display tests.

  1. Create a new CLI option displayOptions=$PWD/tests/reportDisplay.js
  2. This file contains a list of Suite titles that are in turn going to be sections in the conformance part of the report
  3. Each entry in the reports Array should look like this:
    export default [
    {
    suite: 'Data Integrity - Ed25519Signature2020',
    labels: {column: 'Issuer', row: 'Test'},
    ids: {
      // these can be filled in later
      rows: [],
      // these can be filled in later or supplied from implementations
      columns: []
    },
    template: 'matrix'
    }
    ]
  4. This then maps to a Suite formatted like this:
    describe('Data Integrity - Ed25519Signature2020', function() {
    for(const [name, implementation] of implementations) {
    describe(implementationName, async function() {
      it('MUST have this property', async function() {
         // ...stuff happens here
      })
    })
    }
    })
  5. This removes the need for meta-data inside of the tests itself and separate test logic from display logic
  6. If the reporter merges Suites from the display file by title then we can run multiple suites in a single loop or even have a single report run in multiple files aka parallel.

Other Goals:

  1. Switch over passing in packages with a full set of templates instead of individual templates
  2. Make the test stats their own template instead of being a part of body
aljones15 commented 2 years ago

Mocha has provided a high level overview of how their parallelization works as of Mocha ^10 here: https://mochajs.org/#parallel-mode

They also provide these instructions for porting reporters to parallel: https://mochajs.org/#parallel-tests