Multiple & Parallel Provisioning | Optimization to Stress Jenkins Agent Less

[caliskanugur]

Currently, upgrade HA and recurring runs need multiple provisionings at the same time, the provisionings depend on the given clusters matrix. And triggers multiple docker runs:

                   jobs["$it-provisioning"] = { 
                     stage("Parallel Provisioning: $it"){
                       def containerName = "${job_name}${env.BUILD_NUMBER}_$it-provisioning"
                       try {
                         sh "docker run --name ${containerName} -t -e CATTLE_TEST_CONFIG=\"${absoluteConfigPath}\" " +
                       "${imageName} sh -c \"/root/go/bin/gotestsum --format standard-verbose --packages=\"${testsDir}${testPackage}\" --junitfile ${testResultsOut} -- ${testCase} -timeout=\"60m\" -v\""

The reason we don't rely on sub-jobs at this stage is that we need the configuration files that these tests use - If a provisioning test successfully completes, it overwrites the starting configuration to update the cluster name to be used for the next stages.

This ticket aims to optimize this stage by relying on Jenkins executor and spreading these parallel jobs on sub-jobs rather than forcing a pipeline and agent X amount docker runs.

The potential solution for this:

1. Use go provisioning as a sub-job - to spread the load instead of go freeform
2. Copy the patched configuration file back from the sub-job to the main job via the copy artifact plugin

[caliskanugur]

Assigning this ticket to myself because https://github.com/rancher/qa-tasks/issues/712 (7th item) is also related to this ticket's (1st item)