For instance, parameters like headless=true|false, os=linux|windows, java=8|11|17, timezone=..., coverage=collect|ignore are hard to configure manually as it would result in a big number of CI jobs causing long CI durations or it would be hand-picked selection which might miss bugs.
I suggest using a randomized approach, so every time CI uses a slightly different set of parameters, so eventually it would get better coverage.
Note: there can be explicit constraints like "there should be at least one job with Java 8", "at least one job with Linux", "at least one job with Windows", "at least one with headless=true/false", and it could add a random selection of the jobs.
Describe the solution you'd like
Use https://github.com/vlsi/github-actions-random-matrix for GitHub Actions matrix generation
For instance, parameters like
headless=true|false,os=linux|windows,java=8|11|17,timezone=...,coverage=collect|ignoreare hard to configure manually as it would result in a big number of CI jobs causing long CI durations or it would be hand-picked selection which might miss bugs.I suggest using a randomized approach, so every time CI uses a slightly different set of parameters, so eventually it would get better coverage.
Note: there can be explicit constraints like "there should be at least one job with Java 8", "at least one job with Linux", "at least one job with Windows", "at least one with headless=true/false", and it could add a random selection of the jobs.
WDYT?
Are there any workarounds?
The matrix has to be configured manually.
Additional context
Sample integrations: https://github.com/cbeust/testng/actions/runs/1863017796 https://github.com/apache/jmeter/pull/693 https://github.com/qos-ch/logback/pull/556 see also https://github.com/vlsi/github-actions-random-matrix#sample-integrations