Closed dduportal closed 1 year ago
BTW one change I am thinking about is to skip building master
by default (only when a release is planned), so that the builds would at least be limited to PRs.
Ok, let's try the new node pool: https://github.com/jenkinsci/bom/pull/1969
Following the "experiments" in https://github.com/jenkinsci/bom/pull/1969, it seems that there is no "easy & obvious" solution, scoped in the node pool sizing, for the AWS cost decrease. So let's deliver the "split bom and plugin resources" using the same sizing as today and we'll continue to diagnose.
Reminder of the expected benefits:
bom
builds will be easier (separate namespace, separate node pool, etc.).Proposed implementation:
Ensure there are 2 different namespaces in cik8s
:
jenkins-agents-bom
for the "production" bom buildsjenkins-agents-experiments
for the "tests" with bigger node pool (and future experiments)Keep using the cik8s-bom
Kubernetes cloud defined in ci.jenkins.io to provide the "admin defined" pod templates
bom
pipelineOpen a new PR in the bom
project to "only" split resources and rename the current https://github.com/jenkinsci/bom/pull/1969
experiment
namespaceexperiment
kube cloudThis issue surfaced an issue in the cik8s
autoscaler setup: https://github.com/jenkins-infra/aws/pull/405
Let's try a real PR with https://github.com/jenkinsci/bom/pull/2032
Looks good: plugins are able to be built as seen durin the (long) week end.
Why
The
bom
builds are challenging for the infrastructure:What
This issue tracks the work related to using a dedicated node pool(s) designed to run only the
bom
builds to:bom
to wait 1-2 hours for an agent allocation because there is a peak ofbom
buildsbom
build usage- to decrease the cost impactFirst round is to focus only on AWS: no more
bom
builds on DigitalOcean:bom
to AWS and others on DigitalOceanThe sizing for the new node pool follows this principles:
A
bom
build is estimated to run:We do not pay for "agent minutes", but rather for "VM minutes": the more time each builds takes waiting with the node pool fully scaled, the more we pay
Costs report: https://docs.google.com/spreadsheets/d/1_C0I0jE-X0e0vDcdKOFIWcnwpOqWC8RQ4YOCgXNnplY/edit#gid=292621391
How
First tentative:
taints
(to ensure the current workload cannot be run accidentally) to split physicallyjenkins-agents-bom
incik8s
to split logicallypodTemplate()
pipeline method in jenkinsci/bom to improve visibility