jackjia-ibm
commented
3 years ago A little background about where we are regarding parallel testing:
- I implemented the runner designed for parallel testing https://github.com/zowe/zowe-install-packaging/blob/master/tests/installation/src/worker-group-runner.ts but disabled by default https://github.com/zowe/zowe-install-packaging/blob/master/tests/installation/jest.config.js#L16
- The runner will check test annotation (for example https://github.com/zowe/zowe-install-packaging/blob/master/tests/installation/src/__tests__/extended/security-systems/convenience/acf2.ts#L21) to decide which worker to use. The tests on same worker will be queued, and with different worker will be parallel.
- The runner works fine. I remember the issue was about when the parallel test reaches sanity test stage, all workers sanity test put reports in same folder. The sanity test folder is defined here https://github.com/zowe/zowe-install-packaging/blob/master/tests/sanity/test/multi-reporters.json.
- To solve the problem, I was thinking to find a way to separate sanity test report workspace, each worker should have their own workspace and then later the report can be aggregated to central place (https://github.com/zowe/zowe-install-packaging/blob/master/tests/installation/src/merge-test-reports.ts). So basically we need to tell mocha to use customized report folder and overwrite default values in https://github.com/zowe/zowe-install-packaging/blob/master/tests/sanity/test/multi-reporters.json.
MarkAckert
FlappiTomic
Currently with the increase in configurations we are trying to test our whole pipeline takes ~5.5hours and even the nightly takes >3hours as it runs the convenience tests then fmid install and ptf tests on all 3 security systems.
As the 3 LPARs are independent we could run install tests against these systems in parallel, rather than sequentially. In order to not get reporting and container clashes we'd need to separate out the current jobs and trigger all 3 in parallel as separate jobs then analyse the results together.
Maybe a good starting point would be to modify promote nightly to do this separation and then considering forking it (or providing config options to branch) to add the other tests in and merging it with the release candidate testing which runs all the tests. After this we could even create a 'quick BVT' varient that people are required to run on PRs before getting them reviewed, to save on manually waiting and copying builds into different test runs?