Scaling executors by node hardware

[GavinBurris42]

One new method of executor scaling has been added: Scaling by Node Hardware. The options to scale are by vCPU or memory, allowing the user to better allocate node resources based on instance type. The user will also decide on a scaling factor determining the number of executors to be allocated per number of resources. For instance if executors are being scaled by vCPUs with a scaling factor of 2, then 1 executor will be allocated for every 2 vCPUs for that nodes instance type. The lower number of executors will be provisioned to prevent over-provisioning. The user can decide to ignore the scaling by hardware for one or more of the resources by setting its scaling factor to 0. The user will be able to select this method of scaling in a dropdown selection in the Jenkins UI, which also includes the existing method of scaling by instance weight and a no scaling option.

• The "Scale by weights" dropdown option replaces the previous "Scale Executors by Weight" checkbox
• CasC is supported for the the new executor scaling feature under the new field "executorScaler"
• The executor scaler dropdown is not supported for Fleet Label Cloud

Testing done

Unit tests were written to check every case of input for the new dropdown menu, as well as varying cases of inputs for the new Node Hardware selection.

Manual Testing:

The following cases need to be tested for the node hardware selection:

• Non-zero numbers set for vCPU and memory
• 0 is set for both vCPU and memory
• 0 is set for vCPU and not memory
• 0 is set for memory and not vCPU

Steps to take before testing:

• Created an EC2 Launch Template using an Amazon Linux 2 AMI.

• Used the Launch Template to create an ASG.

• Non-zero numbers set for vCPU and memory:

  • Ran the Jenkins EC2-Plugin. Configured the cloud to use the newly created ASG, set the number of executors to 1 and the "Scale Executors" dropdown to "Scale by node hardware". Set vCPU(s) Per Executor to 2 and Memory(GiB) Per Executor to 2.

  • 

  • Ran a test job to have the plugin provision instances, and assure that jobs can be ran successfully.

  • 

  • There are 4 executors provisioned. This is verified by checking the nodes hardware

  • Use the AWS console to check the number of vCPUs and memory. Check the type of instance used, and then look up this instance using the Instance Type tool.

  • 
  • 

  • Calculate number of executors for vCPUs to be 8/2 = 4. Calculate number of executors for memory to be 15/2 = 7. Since 4 < 7, 4 executors have been successfully provisioned.

• 0 is set for both vCPU and memory:

  • Ran the Jenkins EC2-Plugin. Configured the cloud to use the newly created ASG, set the number of executors to 2 and the "Scale Executors" dropdown to "Scale by node hardware". Set vCPU(s) Per Executor to 0 and Memory(GiB) Per Executor to 0.
  • Ran a test job to have the plugin provision instances, and assure that jobs can be ran successfully.
  • Two executors should be provisioned since no scaling by node hardware occurs, and the set number of executors is used.

• 0 is set for vCPU and not memory:

  • Ran the Jenkins EC2-Plugin. Configured the cloud to use the newly created ASG, set the number of executors to 1 and the "Scale Executors" dropdown to "Scale by node hardware". Set vCPU(s) Per Executor to 0 and Memory(GiB) Per Executor to 3.
  • Ran a test job to have the plugin provision instances, and assure that jobs can be ran successfully.
  • There are 5 executors provisioned. This is verified by checking the nodes hardware
  • Use the AWS console to check the number of vCPUs and memory. Check the type of instance used, and then look up this instance using the Instance Type tool. In this case a c4.2xlarge was used with 8 vCPUs and 15GiB of memory.
  • Ignore vCPUs since it is set to 0. Calculate number of executors for memory to be 15/3 = 5. 5 executors have been successfully provisioned.

• 0 is set for memory and not vCPU:

  • Ran the Jenkins EC2-Plugin. Configured the cloud to use the newly created ASG, set the number of executors to 1 and the "Scale Executors" dropdown to "Scale by node hardware". Set vCPU(s) Per Executor to 4 and Memory(GiB) Per Executor to 0.
  • Ran a test job to have the plugin provision instances, and assure that jobs can be ran successfully.
  • There are 2 executors provisioned. This is verified by checking the nodes hardware
  • Use the AWS console to check the number of vCPUs and memory. Check the type of instance used, and then look up this instance using the Instance Type tool. In this case a c4.2xlarge was used with 8 vCPUs and 15GiB of memory.
  • Ignore vCPUs since it is set to 0. Calculate number of executors for memory to be 8/4 = 2. 2 executors have been successfully provisioned.

[pdk27]

@GavinBurris42 , looks like the only action item is to test that the configuration loads accurately in Jenkins UI. Other than that, PR looks good.

[vineeth-bandi]

Approved pending documentation changes for CasC Usage.