.gitlab: directory structure rework

[wihobbs]

Problem: the .gitlab-ci.yml top-level file will become overgrown as we progress in the number of tests that we want to run on our framework repositories. Adding jobs, machines, and tests should be organized and easy in syntax/file structure.

Utilize YAML syntax and include statements to rework the files in a directory structure. Note that the use of include eliminates the functionality of YAML anchors but reference tags are a good workaround.

[wihobbs]

Things left to ponder:

• .standard-variables should probably be split off into variables required in the TOSS/jacamar environment (LLNL_SERVICE_USER, CUSTOM_CI_BUILDS_DIR) and variables required for sharness logging (debug, FLUX_TEST_LOGFILE). HWLOC_COMPONENTS needs to be a machine specific variable as Lassen doesn't have x86.
• As other framework repositories get added to the testing, do they need their own file, i.e. build-sched.gitlab-ci.yml? That begs a directory if so. Or they can all go in the same file, builds.gitlab-ci.yml.
• I feel like a lot of code gets reproduced running these tests on four systems -- adding a new system requires naming tests for that system, and this should be easier. This may be constrained by the way gitlab works, but maybe there's a way to define a list of machines that should run all the tests...

Note that test jobs can only be defined in the top-level .gitlab-ci.yml.

[grondo]

One other thing to ponder:

• I wonder if long term placing all the logic in the gitlab yaml will be the right approach. It is much easier to edit/extend a script (and test that script outside of a gitlab runner) than modifying and testing gitlab yaml directly.

[wihobbs]

I wonder if long term placing all the logic in the gitlab yaml will be the right approach. It is much easier to edit/extend a script (and test that script outside of a gitlab runner) than modifying and testing gitlab yaml directly.

This is a good point, but I'm not sure we've hit this quite yet. The scripts themselves aren't really the culprit of the complexity and length of our .gitlab-ci.yml file -- the complexity is because of the number of machines we're testing and the individual configuration options needed to make them happy. (And all of that always has to be Gitlab sadly.)

But I definitely think we should keep this in mind for MPI testing, because that could script could be much lengthier and require more frequent changes/additions than builds.

[wihobbs]

Thanks! Will merge. I will also push this up to our CZ GitLab repository directly using my LC credentials, since mirroring is currently disabled.