CMCDragonkai
commented
2 years ago I believe we can reuse the child pipeline script generator, which would end up dynamically generating N number of parallel jobs.
It's also possible that child pipelines won't be necessary, and instead we can just use the parallel keyword too https://about.gitlab.com/blog/2021/01/20/using-run-parallel-jobs/.
However I quite like the child pipeline script, it should be useful for any kind of dynamic pipeline creation. Perhaps something to quickly calculate/estimate what the N should be.
Specification
Since we are expanding our testing to include other platforms like macos and windows, this creates an issue for VERY large tests that we have in PK.
In PK we are spinning up a test job for each subdirectory under
tests, this is not efficient under the 10 minute startup routines for windows and mac runners. This is also done via child pipelines, which relies on a script that generates the child pipeline jobs.The only way to do this would be to do test load balancing instead. This is where we create N number of linux, mac, windows runners, and then distribute tests across the N runners. The N number is determined based a tradeoff between fastest pipeline completion time vs total minutes being used. That is we want highest possible pipeline completion with the lowest total minutes being used. The intersection between the 2 lines is the sweet spot. This would have to be figured out manually & iteratively. The test suite here is very small though, so this should be done on PK instead.
However upon using load balancing, we lose track of individual jobs tackling test directories. Right now we are able to see that only certain areas fail, while other areas succeed. To regain this ability, we have to use the test reporters (which is currently integrated already as
junit). There are junit options that can be utilised to maximise the utility of these reports:classNameTemplate: '{classname}'- sets thesuitecolumn on gitlab to the name of the top-level describe of the test filetitleTemplate: '{title}'- sets thenamefield on gitlab to the name of the testaddFileAttribute: 'true'- adds the filename to the junit reportsincludeConsoleOutput: 'true'- includes console output in the reportOur test reporters now can tell us which test/tests are failing without us having to dig through the logs.
But jest is not capturing STDOUT or STDERR anymore.
The alternative is to change our
loggerto use theconsole.errorinstead, which is intercepted by jest and buffered up and should be shown to the test reporters. TheincludeConsoleOutoutoption only affects stdout though, so we need a way to work around this.The buffering causes an issue during development though, it's nice to be able to see the logs immediately when we are debugging test issues, especially for the networking code where the ordering matters.
If we can get jest to unbuffer the logs, or change back to
process.stderrwhen the--ciflag is not set, then we can see the logs unbuffered when running the tests. Right now the--ciflag is set when running the tests in the CI/CD.Finally within our test runners, because each runner only 1 CPU, we can use
--runInBandto keep it simple.Additional context
--shardTasks