proposal for a multi server tests report, with MUST, SHOULD, MAY ...

[bourgeoa]

@michielbdejong @ylebre @edwardsph This is a proposal using MUST, SHOULD, MAY, as described in proposal issue #110 The example below use local repositories, located in solidos/workspaces

• In each tests repository : create a separate run file (run-against-nss.sh, run-against-css.sh, ....) for each server with a jest outputFile in a folder (example bellow -- if running the tests from solidos stack, the result folder will be in workspaces/test-suite/NSS, CSS, ESS, ... -- env values can be used to include/exclude tests

  
  # init section
  export RESULTS_PATH=.../test-suite/NSS/crud-results.json

select section (

export INCLUDE_MAY=1 # by default 'MAY' is excluded export SKIP_SHOULD_A12=1 # skip tests referenced 'SHOULD', 'A12'

npm run jest "$@"

npm run jest -- --verbose --json --outputFile="$RESULTS_PATH" "$@" # verbose is optional

- in test-suite repo run a report builder using all `folder/files` created : example https://github.com/bourgeoa/test-suite/blob/main/test-suite-report.js
- and this is a `simplistic` example text report for the test-suite : https://github.com/bourgeoa/test-suite/blob/main/test-suite-report.txt (the values are real from 8/03/2021)
-- default is `SHOULD` and test numbering default is `Txxx` to link to errors
-- The `MAY` example includes a test id

[edwardsph]

When I started experimenting with this idea I had a script that aggregated the different results. I'll have a look at it tomorrow and see if it's worth you looking at but I suspect it was a minimal placeholder.