Concourse worker job distribution fix

[jackstockley89]

Background

Job are not getting spread evenly across all the workers

Approach

Look into concourse configuration for container placement to fix this https://concourse-ci.org/container-placement.html#container-placement

one of the parameters is fewest-build-containers this could help to make sure jobs are placed on the worker with jobs. This looks like it needs to be added tot he web environments variables

concourse module: templates/values.yaml

web:
  ## Configure resource requests and limits.
  ## Ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ##
  resources:
    requests:
      cpu: "20m"
      memory: "100Mi"
    limits:
      cpu: "100m"
      memory: "250Mi"

  env:
    - name: CONCOURSE_LOG_LEVEL
      value: "debug"

======== add the following =========
    - name: CONCOURSE_CONTAINER_PLACEMENT_STRATEGY
      value: "fewest-build-containers"
==============================

Which part of the user docs does this impact

Communicate changes

• [ ] post for #cloud-platform-update
• [ ] Weeknotes item
• [ ] Show the Thing/P&A All Hands/User CoP
• [ ] Announcements channel

Definition of done

• [ ] another team member has reviewed
• [ ] smoke tests are green

Reference

How to write good user stories

[tmahmood72]

This ticket should be looked at in the same Sprint as ticket #5837

[kyphutruong]

Working branch here