[ci.jenkins.io] separate container agent resources between `bom` and other builds

[dduportal]

Why

The bom builds are challenging for the infrastructure:

• They are slow (ref. https://github.com/jenkinsci/bom/pull/1962#issuecomment-1508783416) and slowing down contributors
• The have an impact on the cloud costs:
  • Compute time: https://github.com/jenkins-infra/helpdesk/issues/3502 and https://github.com/jenkins-infra/helpdesk/issues/3487
  • Bandwidth: https://github.com/jenkins-infra/helpdesk/issues/3485
• They are quite often unreliable and need to be retried: https://github.com/jenkinsci/bom/pull/1962#issuecomment-1508783416

What

This issue tracks the work related to using a dedicated node pool(s) designed to run only the bom builds to:

• Avoid contributors of other projects than the bom to wait 1-2 hours for an agent allocation because there is a peak of bom builds
• Design the resource allocation specifically for the bom build usage- to decrease the cost impact

First round is to focus only on AWS: no more bom builds on DigitalOcean:

• DigitalOcean credits should be preserved (https://github.com/jenkins-infra/helpdesk/issues/3487)
• If AWS costs does not decrease, then we can emove the "normal" node pool and split bom to AWS and others on DigitalOcean
• Simpler baseline for metrics and cost analysis: only 1 kind of builds

The sizing for the new node pool follows this principles:

• A bom build is estimated to run:

  • Between 200 and 300 parallel steps: we'll count ~300 elements in the build queue as baseline
  • Each step takes ~ 10-11 minutes (used to be ~4-5 before https://github.com/jenkinsci/bom/pull/1955 but outbound bandwidth costs way more)
  • Each step is retried up to 3 times when the underlying agent is killed (due to resources: OOM, CPU limit, eviction, spot instance reclaim), which had way more elements in the build queue

• We do not pay for "agent minutes", but rather for "VM minutes": the more time each builds takes waiting with the node pool fully scaled, the more we pay

  • Massively parallelizing to take the build queue item is a priority to decrease both costs and pressure on ci.jenkins.io

Costs report: https://docs.google.com/spreadsheets/d/1_C0I0jE-X0e0vDcdKOFIWcnwpOqWC8RQ4YOCgXNnplY/edit#gid=292621391

How

First tentative:

• A new node pool with taints (to ensure the current workload cannot be run accidentally) to split physically
  • Keeping spot instances for now, to avoid cost explosion. But ensure that the instance spot costs allow up to 4 retries before reaching the on-demand (non-spot) cost (below the 3 retries of the pipeline)
  • Decrease the "waste" of resources in the instances: more pods per node, right sizing (room for improvement for second tentative)
  • Increase the maximum number of parallel pods to ensure a given bom builds can be treated in "one shot"
• A new namespace jenkins-agents-bom in cik8s to split logically
• Use the podTemplate() pipeline method in jenkinsci/bom to improve visibility

[dduportal]

• Node pool initial implementation: https://github.com/jenkins-infra/aws/pull/388

[dduportal]

• Adding the new namespace: https://github.com/jenkins-infra/kubernetes-management/pull/3846

[dduportal]

• https://github.com/jenkins-infra/helm-charts/pull/488 along the way (update Service Accounts behavior after Kubernetes 1.23 to have an explicit API token)
  • Deployed in https://github.com/jenkins-infra/kubernetes-management/pull/3847
• https://github.com/jenkins-infra/jenkins-infra/pull/2770 to add a new Kubernetes Cloud on ci.jenkins.io, tested with success to spawn agents on the new node pool

[jglick]

BTW one change I am thinking about is to skip building master by default (only when a release is planned), so that the builds would at least be limited to PRs.

[dduportal]

Ok, let's try the new node pool: https://github.com/jenkinsci/bom/pull/1969

[dduportal]

• Configuration fixup to ensure datadog agent runs on the new nodepool (toleration): https://github.com/jenkins-infra/kubernetes-management/pull/3850

[dduportal]

Following the "experiments" in https://github.com/jenkinsci/bom/pull/1969, it seems that there is no "easy & obvious" solution, scoped in the node pool sizing, for the AWS cost decrease. So let's deliver the "split bom and plugin resources" using the same sizing as today and we'll continue to diagnose.

Reminder of the expected benefits:

• Non bom builds (plugins, packaging, etcc.) using Linux container agents on ci.jenkins.io won't have to wait 1-2 hours when there are bom builds waiting.
• Observability of the bom builds will be easier (separate namespace, separate node pool, etc.).

Proposed implementation:

• Ensure there are 2 different namespaces in cik8s:

  • jenkins-agents-bom for the "production" bom builds
  • jenkins-agents-experiments for the "tests" with bigger node pool (and future experiments)

• Keep using the cik8s-bom Kubernetes cloud defined in ci.jenkins.io to provide the "admin defined" pod templates

  • Expecting 2 pod templates, one per namespace with each one its own "max pod limit"
  • Defaults to JDK11 agent and JDK17 for dev tools. Specific Java version for dev tools will be defined in the bom pipeline

• Open a new PR in the bom project to "only" split resources and rename the current https://github.com/jenkinsci/bom/pull/1969

[dduportal]

• [x] New node pool: https://github.com/jenkins-infra/aws/pull/396
• [x] New namespace: https://github.com/jenkins-infra/kubernetes-management/pull/3888
• Update ci.jenkins.io agent configuration (https://github.com/jenkins-infra/jenkins-infra/pull/2805)
  • [x] insert new SA token for experiment namespace
  • [x] Add experiment kube cloud
  • [x] Update limits
• [x] Update updatecli manifest for namespace quotas in kubernetes management - https://github.com/jenkins-infra/kubernetes-management/pull/3889
• [x] Validate automated updatecli PR updating pods quota in kubernetes management - https://github.com/jenkins-infra/kubernetes-management/pull/3890
• [x] Update jenkins-infra/jenkins-infra to define agent template scheduling data (nodeselector and tolerations) - https://github.com/jenkins-infra/jenkins-infra/pull/2808

[dduportal]

This issue surfaced an issue in the cik8s autoscaler setup: https://github.com/jenkins-infra/aws/pull/405

[dduportal]

Let's try a real PR with https://github.com/jenkinsci/bom/pull/2032

[dduportal]

Looks good: plugins are able to be built as seen durin the (long) week end.