Jenkins: Change pipeline durability level

[AdamBrousseau]

Doc: https://jenkins.io/doc/book/pipeline/scaling-pipeline/

There are 3 ways to configure the durability setting:

1. Globally, you can choose a global default durability setting under "Manage Jenkins" > "Configure System", labelled "Pipeline Speed/Durability Settings". You can override these with the more specific settings below.
2. Per pipeline job: at the top of the job configuration, labelled "Custom Pipeline Speed/Durability Level" - this overrides the global setting. Or, use a "properties" step - the setting will apply to the NEXT run after the step is executed (same result).
3. Per branch for a Multibranch project. (We don't have this project type)

What Are The Durability Settings?
1. Performance-optimized mode ("PERFORMANCE_OPTIMIZED") - Greatly reduces disk I/O. If Pipelines do not finish AND Jenkins is not shut down gracefully, they may lose data and behave like Freestyle projects — see details above.
2. Maximum durability ("MAX_SURVIVABILITY") - behaves just like Pipeline did before, slowest option. Use this for running your most critical Pipelines.
3. Less durable, a bit faster ("SURVIVABLE_NONATOMIC") - Writes data with every step but avoids atomic writes. This is faster than maximum durability mode, especially on networked filesystems. It carries a small extra risk (details above under "What Am I Giving Up: Atomic Writes").

Jenkins pipelines are by default, high-durability, low performance. If your jobs are mostly spent doing a few commands, then this default mode is not advantageous. Since >90% of our job time is spent running a single command (i.e. make compile or make 'test') this mode is not a good fit for us. If our jobs were 100's of smaller, incremental 'steps' this would be a better option becasue it allows pipelines to resume after an unexpected shutdown. Since most/all of our pipelines are similar in nature, the global setting is likely the easiest change to make.

Edit: This will likely benefit the 'check' PR jobs the most as these perform several small operations. It will also likely improve the perf of Jenkins overall as master will have much less I/O happening, particularly when the farm is very loaded.

Note: Individual jobs can still override the global setting in their config. They can also individually select to not allow pipelines to resume after restart which will also change the durability level to performance optimized.

[AdamBrousseau]

Changed to global setting to Performance optimized

There is an obvious difference in speed among 'steps' Ex. Build job - Running on master for initial setup Before: 3min40

00:11:05.447 Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/Build-JDK10-aix_ppc-64_cmprssptrs/workspace
[Pipeline] {
[Pipeline] retry
[Pipeline] {
[Pipeline] checkout
00:11:21.455 Cloning the remote Git repository
00:11:21.455 Cloning repository https://github.com/eclipse/openj9.git
00:11:21.455  > git init /home/hudson/genie.openj9/.jenkins/jobs/Build-JDK10-aix_ppc-64_cmprssptrs/workspace # timeout=10
00:11:24.287 Fetching upstream changes from https://github.com/eclipse/openj9.git
00:11:24.287  > git --version # timeout=10
00:11:24.293  > git fetch --tags --progress https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
00:11:31.924  > git config remote.origin.url https://github.com/eclipse/openj9.git # timeout=10
00:11:34.372  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
00:11:34.960  > git config remote.origin.url https://github.com/eclipse/openj9.git # timeout=10
00:11:34.966 Fetching upstream changes from https://github.com/eclipse/openj9.git
00:11:34.966  > git fetch --tags --progress https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
00:11:46.443  > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
00:11:46.450  > git rev-parse refs/remotes/origin/refs/heads/master^{commit} # timeout=10
00:11:46.666 Checking out Revision da56ab5e9a8a0e475b8e74936fdf4601e8f28e69 (refs/remotes/origin/master)
00:11:46.667  > git config core.sparsecheckout # timeout=10
00:11:46.675  > git checkout -f da56ab5e9a8a0e475b8e74936fdf4601e8f28e69
00:11:50.214 Commit message: "Merge pull request #2173 from pdbain-ibm/attach"
00:11:51.921  > git rev-list --no-walk da56ab5e9a8a0e475b8e74936fdf4601e8f28e69 # timeout=10
[Pipeline] }
[Pipeline] // retry
[Pipeline] load
[Pipeline] { (buildenv/jenkins/common/variables-functions)
[Pipeline] }
[Pipeline] // load
[Pipeline] fileExists
[Pipeline] echo
00:12:54.107 Using variables file: buildenv/jenkins/variables/defaults.yml
[Pipeline] readYaml
[Pipeline] echo
00:12:58.714 Using OPENJDK_REPO = https://github.com/ibmruntimes/openj9-openjdk-jdk10.git OPENJDK_BRANCH = openj9 OPENJDK_SHA = 3e423891781db77e0ae7b7371c611c4d5af0d28b
[Pipeline] echo
00:13:11.958 Using OPENJ9_REPO = https://github.com/eclipse/openj9.git OPENJ9_BRANCH = master OPENJ9_SHA = da56ab5e9a8a0e475b8e74936fdf4601e8f28e69
[Pipeline] echo
00:13:15.563 Using OMR_REPO = https://github.com/eclipse/openj9-omr.git OMR_BRANCH = openj9 OMR_SHA = da0c7eeba658603d533882bca909f1d5e576a251
[Pipeline] load
[Pipeline] { (buildenv/jenkins/common/build)
[Pipeline] }
[Pipeline] // load
[Pipeline] cleanWs
00:13:45.476 [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done

After: 17 seconds

00:03:48.335 Running on Jenkins in /home/hudson/genie.openj9/.jenkins/jobs/Build-JDK10-aix_ppc-64_cmprssptrs/workspace
[Pipeline] {
[Pipeline] retry
[Pipeline] {
[Pipeline] checkout
00:03:48.380 Cloning the remote Git repository
00:03:48.380 Cloning repository https://github.com/eclipse/openj9.git
00:03:48.380  > git init /home/hudson/genie.openj9/.jenkins/jobs/Build-JDK10-aix_ppc-64_cmprssptrs/workspace # timeout=10
00:03:48.390 Fetching upstream changes from https://github.com/eclipse/openj9.git
00:03:48.390  > git --version # timeout=10
00:03:48.395  > git fetch --tags --progress https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
00:04:02.894  > git config remote.origin.url https://github.com/eclipse/openj9.git # timeout=10
00:04:02.900  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
00:04:02.906  > git config remote.origin.url https://github.com/eclipse/openj9.git # timeout=10
00:04:02.911 Fetching upstream changes from https://github.com/eclipse/openj9.git
00:04:02.911  > git fetch --tags --progress https://github.com/eclipse/openj9.git +refs/heads/*:refs/remotes/origin/*
00:04:03.188  > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
00:04:03.194  > git rev-parse refs/remotes/origin/refs/heads/master^{commit} # timeout=10
00:04:03.201 Checking out Revision e27ef8ecb42e66292f92bcdabdca7f959261354d (refs/remotes/origin/master)
00:04:03.201  > git config core.sparsecheckout # timeout=10
00:04:03.207  > git checkout -f e27ef8ecb42e66292f92bcdabdca7f959261354d
00:04:04.589 Commit message: "Merge pull request #2195 from hangshao0/sharedCache"
00:04:04.590  > git rev-list --no-walk da56ab5e9a8a0e475b8e74936fdf4601e8f28e69 # timeout=10
[Pipeline] }
[Pipeline] // retry
[Pipeline] load
[Pipeline] { (buildenv/jenkins/common/variables-functions)
[Pipeline] }
[Pipeline] // load
[Pipeline] fileExists
[Pipeline] echo
00:04:05.239 Using variables file: buildenv/jenkins/variables/defaults.yml
[Pipeline] readYaml
[Pipeline] echo
00:04:05.263 Using OPENJDK_REPO = https://github.com/ibmruntimes/openj9-openjdk-jdk10.git OPENJDK_BRANCH = openj9 OPENJDK_SHA = 3e423891781db77e0ae7b7371c611c4d5af0d28b
[Pipeline] echo
00:04:05.265 Using OPENJ9_REPO = https://github.com/eclipse/openj9.git OPENJ9_BRANCH = master OPENJ9_SHA = e27ef8ecb42e66292f92bcdabdca7f959261354d
[Pipeline] echo
00:04:05.269 Using OMR_REPO = https://github.com/eclipse/openj9-omr.git OMR_BRANCH = openj9 OMR_SHA = da0c7eeba658603d533882bca909f1d5e576a251
[Pipeline] load
[Pipeline] { (buildenv/jenkins/common/build)
[Pipeline] }
[Pipeline] // load
[Pipeline] cleanWs
00:04:05.569 [WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done

You can also see in the after case how much sooner the job ran on the master.

Ex 2. Reference repo update and workspace cleanup. This job runs a set of ~15 'steps' on all nodes in parallel. I ran 1 of these initially then 2 more with the default and changed setting. The first one would have done any update and cleanup leaving the next 2 to basically do no work other than executing the commands. I also ran them when the farm was idle. Before: 11min After: 3min24

[DanHeidinga]

Are you suggesting switching to PERFORMANCE_OPTIMIZED mode and accepting the risk that updates may be lost if the pipeline fails and jenkins is shutdown?

We get slightly faster builds from this and put less IO load on the server, making us better jenkins citizens?

@AdamBrousseau Is that an accurate summary? If so, sounds like a good tradeoff to me.

[AdamBrousseau]

Not sure what you mean by

risk that updates may be lost if the pipeline fails

But yes, that is a fair summary. It's obviously not going to speed up the "work" of a build (make compile for example) but it should speed up all the in between bits. And yes, I'm thinking and hoping the load on Master will be greatly reduced during high farm load.

Edit: Basically if Jenkins unexpectedly shuts down, anything running will have to start over from he beginning.

[DanHeidinga]

Edit: Basically if Jenkins unexpectedly shuts down, anything running will have to start over from he beginning.

How often does our Jenkins unexpectedly shut down? If it's infrequent, then I'm OK with this.

[smlambert]

Good suggestion @AdamBrousseau, in my opinion, this is the right thing to do. Given the few times when Jenkins has gone down or been unavailable (even less likely after the artifact 'cleanup'), and that the builds are easily rerun from scratch (our tools/services should always be easy to start/create from scratch, so we can take 'risks' for greater benefit).

[AdamBrousseau]

How often does our Jenkins unexpectedly shut down

I would say not very often. There has been a few instances, some of them our fault. But in general I think it's pretty stable. Also as Shelley mentioned, our stuff is easy to rebuild from scratch in a timely manner.

Since there seems to be a general +1 for this, I'll close the issue as I've already changed the setting and will spam the Slack channels to get a larger awareness.