Builds and Sets of Unit & System Tests Should be Runnable on Chosen OSs Without Using Build Packages From Branch

[thomashampson]

Is your feature request related to a problem? Please describe. Developers often don't have access to all target OS so will routinely use the pull request tests to check whether their code changes build and pass all tests on all OS. This results in:

• slow feedback time: you may want to simply know whether the code compiles, or run a single unit test
• unnecessary load on the build servers: you might only need to check a Windows build, but jobs will run on all OS, causing a build queue on macOS for no reason
• it's also not possible to run the system tests on anything other than Linux: this may result in system test changes breaking the nightly pipeline if they've only been tested on Linux via PR checks.

Describe the solution you'd like A new Jenkins pipeline where the following can be specifiied:

• branch name
• bool for each OS
• whether to run unit tests
• whether to run system tests
• regex for specifying tests to run

[robertapplin]

If possible, it would be useful to have an option to run doc tests as well!

[jhaigh0]

do we see doc test results vary between different linux/windows/mac? I guess they're running python snippets so they could.

[robertapplin]

I have seen results vary between Linux and Mac. Some sphinx warnings have snuck past our Mac jobs building the docs, for it then to fail when built on Linux in the nightly pipeline

[rboston628]

PR #38130 would have been a good use-case for this dev pipeline. There may be a few more instances of wanting to run single system tests on several systems, so this pipeline would be very useful.